Skip to content

feat(llma): support composite offline scorers - #1235

Open
Radu-Raicea wants to merge 1 commit into
mainfrom
feat/llma-composite-scorers
Open

Radu-Raicea wants to merge 1 commit into
mainfrom
feat/llma-composite-scorers

Conversation

@Radu-Raicea

Copy link
Copy Markdown
Member

💡 Motivation and Context

Offline evaluation callers can create and version composite scorers, validate formulas, and request per-item calculations through the existing sync and async clients. Public types expose pinned inputs and dependency metadata; upload receipts expose stored outcomes, including calculation errors.

Bulk uploads keep composite items and their supplied inputs together while preserving order and exact retry recovery. Ordinary uploads retain their existing wire format. Includes documentation, a replayable example, and a minor release changeset.

Requires the backend rollout from PostHog/posthog#132949. The accompanying SDK contract update defines the new behavior; no intentional contract divergence.

💚 How did you test it?

  • Automated tests covering sync/async management, validation, receipts, chunk boundaries, and replay: 652 focused tests pass.
  • Formatting, lint, typing, strict downstream API usage, public API snapshot, and generated mirror import checks pass.
  • The full suite's socket-dependent tests pass outside the sandbox. Three metrics fork tests still crash on this macOS host and also fail against untouched main.

📝 Checklist

  • All commits are signed.
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change.
  • Added a Sampo changeset.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Codex assisted implementation and validation using Git, GitHub, uv, pytest, Ruff, mypy, and Pyright. Composite grouping is conservative because ingestion-only credentials cannot discover dependencies; oversized groups fail before any upload. Validation used mocked API contracts and inspected backend source, without a live-backend calculation test.

@Radu-Raicea
Radu-Raicea requested a review from a team as a code owner November 11, 2026 20:17
@Radu-Raicea Radu-Raicea self-assigned this Nov 11, 2026
@veria-ai

veria-ai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

PR overview

No security issues were found in this pull request.

Reviews

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant