feat(AIC-3363): Support inline datasets

packages/ai/README.md

@@ -73,6 +73,8 @@ result = await evals.run(
 )
 ```
 
+Pass `rows=[DatasetRow(...)]` in place of `dataset` to evaluate rows supplied from code instead of an LD-hosted dataset; `DatasetRow` is re-exported here too, and exactly one of the two arguments is required.
+
 `LD_API_TOKEN` is required. Configure `LD_SDK_KEY` — or initialize your own client with `init_client(client=...)` — to emit one `$ld:ai:offline-evals:generation` event per generated row, plus one `$ld:ai:offline-evals:criterion` event per `(row, criterion)` when `criteria` are supplied, through the standard SDK event transport. The SDK reports scores; LaunchDarkly rules on them at ingest. A judge served by a different provider than `generation` needs a handler for it in `judge_handlers`. Use `LD_API_BASE_URI` for staging or local management API traffic; it is separate from the SDK delivery setting `LD_BASE_URI`. Evaluation-run links use the explicit `ui_base_uri` option or `LD_UI_BASE_URI` (for example, `https://ld-stg.launchdarkly.com` in staging), defaulting to `https://app.launchdarkly.com`. See the [core evaluations guide](../client/README.md#run-an-evaluation-from-code).
 
 ---

packages/client/README.md

@@ -46,7 +46,7 @@ No code changes are required — `init_client()` detects the packages at runtime
 
 ### Run an evaluation from code
 
-The evaluations harness reads an LD-hosted dataset, creates a new evaluation and API-source run, and invokes your handler once per row. Rows can then be scored by LaunchDarkly judges and local scorer functions; see [Score rows with judges and scorers](#score-rows-with-judges-and-scorers). Each success or error queues a `$ld:ai:offline-evals:generation` custom event containing the evaluation, run, dataset, and row identifiers plus output or error (`errorMessage` is included for `ERROR` rows), nested `usage.inputTokens`/`usage.outputTokens`, timing, and stable hashes. Dataset-owned input, expected output, metadata, and variables are not duplicated in the event. Each queued event is logged at `INFO` on the `launchdarkly_ai_server.evaluations.runner` logger with its RFC3339 UTC `emittedAt` timestamp and stable `eventId`, making it possible to compare SDK emission time with ClickHouse arrival time once that logger is enabled. The same `emittedAt` value is included in the event payload. Events are flushed before the summary is fetched and the call returns; handlers are never rerun to retry event delivery. Pass/fail is derived from LaunchDarkly's run summary.
+The evaluations harness takes a dataset, creates a new evaluation and API-source run, and invokes your handler once per row. Rows come from exactly one of two places: pass `dataset` to read an LD-hosted dataset by key, or pass `rows` to supply them from code (see [Supply dataset rows from code](#supply-dataset-rows-from-code)). Rows can then be scored by LaunchDarkly judges and local scorer functions; see [Score rows with judges and scorers](#score-rows-with-judges-and-scorers). Each success or error queues a `$ld:ai:offline-evals:generation` custom event containing the evaluation, run, and row identifiers plus output or error (`errorMessage` is included for `ERROR` rows), nested `usage.inputTokens`/`usage.outputTokens`, timing, and stable hashes. When rows come from an LD-hosted dataset the event carries the dataset identifiers and omits the row's own input, expected output, metadata, and variables, because the dataset already holds them; when you pass `rows` inline the reverse is true — LaunchDarkly has no copy, so the generation event carries `input`, `expectedOutput`, `variables`, and `metadata` and no dataset identifiers. The criterion event never carries row data in either mode. Each queued event is logged at `INFO` on the `launchdarkly_ai_server.evaluations.runner` logger with its RFC3339 UTC `emittedAt` timestamp and stable `eventId`, making it possible to compare SDK emission time with ClickHouse arrival time once that logger is enabled. The same `emittedAt` value is included in the event payload. Events are flushed before the summary is fetched and the call returns; handlers are never rerun to retry event delivery. Pass/fail is derived from LaunchDarkly's run summary.
 
 Result links use `ui_base_uri`, then `LD_UI_BASE_URI`, then `https://app.launchdarkly.com`; this is independent of `LD_API_BASE_URI`. After flushing generation events, the harness polls the run summary endpoint until passed + failed + error rows fully account for a nonzero total with no pending rows, polling every `poll_interval_seconds` (default 2s) up to `poll_timeout_seconds` (default 180s); pass either to `run()` to widen both for large datasets. The summary endpoint does not return run state, so `RunSummary` exposes row counts only. A run passes only when the completed summary has no failed, error, or pending rows. `failed_rows` counts rows whose criteria were scored and did not meet their threshold, so a gate that ignored it would exit 0 on a run where every row failed its judge. Evaluation keys must be unique because every call creates a new evaluation with `POST`.
 
@@ -80,6 +80,37 @@ sys.exit(asyncio.run(main()))
 
 `project_key` is supplied per run rather than during initialization. `generation.instructions` is shorthand for one system message; use `generation.messages` instead for a full message list, but do not supply both. The harness never retries a handler invocation because doing so could repeat tool side effects. Its retries apply only to LaunchDarkly management API requests.
 
+### Supply dataset rows from code
+
+Pass `rows` instead of `dataset` to evaluate rows you already have — a CI fixture list, a JSON file, a generated corpus — without creating a hosted dataset first. The run then reads no dataset and issues no dataset requests.
+
+```python
+from launchdarkly_ai_server import DatasetRow, init_evaluations
+
+result = await init_evaluations().run(
+    project_key="my-project",
+    key="support-qa-2026-08-20",
+    rows=[
+        DatasetRow(
+            row_index=0,
+            input="Where is order {{order_id}}?",
+            expected_output="Order {{order_id}} shipped on the 3rd.",
+            variables={"order_id": "A19"},
+            metadata={"suite": "orders"},
+        ),
+        DatasetRow(row_index=1, input="How do I get a refund?"),
+    ],
+    handler=create_openai_messages_handler(),
+    generation={"provider": "OpenAI", "model": "gpt-4o"},
+)
+```
+
+Exactly one of `dataset` and `rows` is required; supplying both, or neither, raises before any request is made. `input` and `expected_output` are `{{variable}}` templates rendered against the row's `variables` exactly as a hosted dataset's are, and a placeholder with no matching variable is left literal — so data that legitimately contains braces survives. The `DatasetRow` objects you pass are never mutated, so the same list is safe to reuse across runs.
+
+You own `row_index`. LaunchDarkly identifies an inline row by `(run, row_index)`, so the values must be unique within the list; a duplicate would collapse two rows into one stored row and the run could never account for every row. `run()` rejects duplicates, negative or non-integer indices, non-string `input`/`expected_output`, and `variables`/`metadata` whose contents the event transport could not serialize (a date, a set, a non-finite number, a bare object) — all before any record exists. That last check is worth knowing about: the SDK's event transport serializes on a background thread and cannot report a failure back to `run()`, so an unencodable value would otherwise become silently lost events and then a polling timeout with nothing to explain it.
+
+Inline rows are also bounded: at most 10,000 rows per run, with all DatasetRow fields each at most 1 MiB (1,048,576 bytes) once UTF-8 encoded — the encoded length, not the character count, because that is what travels in the event. Both are checked before any record exists, so an oversized dataset costs no API requests and no generation calls. Neither bound applies to a hosted `dataset`: LaunchDarkly holds those rows already, and a caller could not fix an oversized one from their own process. Pass `dataset` instead of `rows` for anything larger.
+
 Generation and criterion events are the only path by which row results reach LaunchDarkly, so `init_evaluations()` raises rather than creating a run that can never complete unless it can resolve an event transport: either an SDK key (`sdk_key` or `LD_SDK_KEY`) or a client already initialized through `init_client(client=...)`. Bringing your own client lets a process emit evaluation events without an SDK key in scope. Every generated row is emitted and flushed unconditionally; no feature flag gates event publishing. The harness then polls the summary endpoint until row accounting shows processing is complete.
 
 ### Score rows with judges and scorers
@@ -112,7 +143,7 @@ result = await init_evaluations().run(
 )
 ```
 
-`Scorer.fn` receives the `DatasetRow` the output was generated from plus the generated output, may be sync or async, and must return a bool or a number from 0 to 1; booleans become 1.0 or 0.0. `Judge.threshold` defaults to 0.5 and `Scorer.threshold` to 1.0 — a perfect score, which is what a boolean scorer wants — and both accept an optional `pass_rate_threshold`. Judge keys and scorer names share one `criterionType` namespace and must be unique within a run, case-insensitively, because that name is part of each result's deterministic event identity. `Judge.ground_truth_context` overrides what the judge is graded against when the dataset row's expected output is not it.
+`Scorer.fn` receives the `DatasetRow` the output was generated from plus the generated output, may be sync or async, and must return a bool or a number from 0 to 1; booleans become 1.0 or 0.0. That row carries the *rendered* `input` and `expected_output`, and its `variables` has been augmented with those two rendered values under the keys `input` and `expected_output` — the same shape whichever source the row came from. `Judge.threshold` defaults to 0.5 and `Scorer.threshold` to 1.0 — a perfect score, which is what a boolean scorer wants — and both accept an optional `pass_rate_threshold`. Judge keys and scorer names share one `criterionType` namespace and must be unique within a run, case-insensitively, because that name is part of each result's deterministic event identity. `Judge.ground_truth_context` overrides what the judge is graded against when the dataset row's expected output is not it.
 
 **The SDK reports scores and never rules on them.** LaunchDarkly derives each row's verdict at ingest by comparing the score against the criterion's stored threshold and success direction, so pass/fail policy is one server-side implementation that applies to every SDK version and to runs already recorded. A judge's direction lives on its AI Config and is injected server-side, keeping the one input a verdict turns on server-attested; a `Scorer` has no LaunchDarkly-side config to read, so it declares its own `success_direction` (default `"higher_is_better"` — set `"lower_is_better"` for a scorer that counts something unwanted, like a regex hit count).
 

packages/client/agents.md

@@ -131,7 +131,9 @@ Handlers may return any of these — the client normalizes them before emitting
 
 `init_evaluations()` creates an evaluations harness using `LD_API_TOKEN` and the management API host `LD_API_BASE_URI`. Do not reuse `LD_BASE_URI`: that variable configures SDK delivery and may point at a relay proxy. Evaluation-run links use the separate `ui_base_uri` option, then `LD_UI_BASE_URI`, then `https://app.launchdarkly.com`; do not derive their host from `LD_API_BASE_URI`. An event transport is resolved in `init_evaluations()`, which raises before any network I/O when it finds neither an SDK key (`sdk_key` or `LD_SDK_KEY`) nor an already-initialized event-capable client: generation events are the only ingest path for row results, so a run without a transport could never complete. The lifecycle module's bring-your-own-client path (`init_client(client=...)`) therefore satisfies the check on its own, and `run()` reuses that singleton through `_resolve_client`; `run()` raises if the client disappears before it emits. Both polling arguments reject NaN, which would otherwise never compare past a deadline and hang the run. The harness always queues one `$ld:ai:offline-evals:generation` custom event per row through the standard SDK event transport and flushes before returning. No feature flag gates event emission. The harness polls the run summary endpoint until a nonzero `total_rows` has `pending_rows == 0` and `passed + failed + error` rows accounting for the total, polling every `poll_interval_seconds` (default 2s) until `poll_timeout_seconds` (default 180s); both are `run()` arguments so large datasets can widen them. The summary endpoint does not return run state, so `RunSummary` exposes row counts only.
 
-`await EvaluationsModule.run(...)` takes `project_key` per call. Dataset lookup/row pagination, evaluation creation, and run creation are private helpers; only `run()` is public. Each call creates a new evaluation with `POST` and a run with `source="api"`, so its key must be unique. The harness directly invokes the supplied handler once per row and never retries it — event delivery is never a reason to rerun a handler because that would repeat tool side effects; retries apply only to management API requests. A 429 is replayed for any method, but 5xx responses and transport failures are replayed only for `GET`/`HEAD`, so an evaluation or run `POST` that the server may already have applied is never duplicated. Management API calls run in a worker thread (`asyncio.to_thread`) because the client is synchronous; the caller's event loop stays free. Generation events go through the already-initialized SDK client when the application has one — `init_client` is idempotent, so an existing singleton wins and the evaluations SDK key is ignored with a warning. Dataset-owned `input`, `expected_output`, `metadata`, and `variables` are deliberately excluded from the event payload. The harness flushes events, polls the run summary endpoint until row accounting is complete (`total_rows > 0`, `pending_rows == 0`, and `passed + failed + error == total_rows`), and raises a timeout once `poll_timeout_seconds` elapses if the backend never reaches one. `RunSummary` includes row counts only, and `EvalRunResult.passed` is true only when error and pending row counts are both zero.
+`await EvaluationsModule.run(...)` takes `project_key` per call, and exactly one of `dataset` (an LD-hosted dataset key) or `rows` (rows supplied from code). Dataset lookup/row pagination, evaluation creation, and run creation are private helpers; only `run()` is public. Each call creates a new evaluation with `POST` and a run with `source="api"`, so its key must be unique. The harness directly invokes the supplied handler once per row and never retries it — event delivery is never a reason to rerun a handler because that would repeat tool side effects; retries apply only to management API requests. A 429 is replayed for any method, but 5xx responses and transport failures are replayed only for `GET`/`HEAD`, so an evaluation or run `POST` that the server may already have applied is never duplicated. Management API calls run in a worker thread (`asyncio.to_thread`) because the client is synchronous; the caller's event loop stays free. Generation events go through the already-initialized SDK client when the application has one — `init_client` is idempotent, so an existing singleton wins and the evaluations SDK key is ignored with a warning.
+
+Which row source a run uses changes exactly three things. A `rows=` run reads no dataset, so it issues neither dataset GET; its run-creation body is exactly `{"source": "api"}` with `datasetId` omitted rather than nulled; and `datasetId`/`datasetKey` drop off both event payloads, which shortens the generation event's identity set from six fields to five and therefore changes its `eventId` (ingest keys such a row off `(run, rowIndex)` alone and never reads `eventId`). Because LaunchDarkly holds no copy of an inline row, the generation event carries its `input`, `expectedOutput`, `variables`, and `metadata` — the one case where those are not excluded. For a `dataset=` run they stay excluded because the dataset owns them, and the criterion event excludes them in both modes. Rendering and the `input`/`expected_output` variable injection go through one shared helper (`_render_row`) for both sources, with `_row_from_api_item` coercing bad server data and `_validate_rows` rejecting bad caller data outright; keep that split rather than unifying it. The `MAX_ROWS` and `MAX_INLINE_TEXT_BYTES` ceilings are inline-only for the same reason: an inline row is only bounded because it travels inside its own generation event, whereas a hosted dataset is LaunchDarkly's to bound and a caller could not shrink one from their process — enforcing the caps there would fail a run over data the caller cannot reach. `MAX_INLINE_TEXT_BYTES` is measured on the UTF-8 encoding of all fields individually, not on `len()` and not on the row as a whole, and on the *rendered* row rather than the caller's: expanding a `{{...}}` placeholder can grow `input` or `expected_output` past the cap, and the injected `input`/`expected_output` keys always grow `variables`, so a row measured before rendering can pass the cap and still produce an event that ingest rejects on the SDK's background flush thread, where nothing can report it and the run only shows up as a polling timeout. That is why `MAX_ROWS` and the shape checks sit in `_validate_rows` while the byte cap sits in `_prepare_inline_rows`, which renders first; `run()` calls it ahead of all I/O so both still report with zero requests issued, and it is the single render per run — the rows it returns are what `_run_rows` and the generation events both use. The harness flushes events, polls the run summary endpoint until row accounting is complete (`total_rows > 0`, `pending_rows == 0`, and `passed + failed + error == total_rows`), and raises a timeout once `poll_timeout_seconds` elapses if the backend never reaches one. `RunSummary` includes row counts only, and `EvalRunResult.passed` is true only when error and pending row counts are both zero.
 
 ---
 
@@ -250,6 +252,14 @@ When `enabled` is `False`, `config` is always `None`. When `enabled` is `True` b
 
 `execute_and_track` expects the handler to return a plain `dict` with at least `output` and `usage` keys. Do not return a custom class — `parse_usage` and the telemetry pipeline both access dict keys.
 
+### 3. Letting an unserializable value into an evaluation event payload
+
+An SDK event buffer is drained and serialized on a background thread, so a value the JSON encoder cannot encode is not reported back to the caller — the events are simply lost, and the run ends in a polling timeout with nothing to explain it. This is only reachable through `run(rows=[...])`, where `variables`/`metadata` hold arbitrary caller objects, which is why `_validate_rows` serialization-checks them up front with `allow_nan=False` and refuses to coerce. Do not relax that into a `default=str` rescue: silently stringifying a caller's value changes what a judge renders and what LaunchDarkly stores, and is unrecoverable once the row is persisted.
+
+### 4. Rendering an evaluation row in place
+
+`DatasetRow` is a mutable dataclass and, for `run(rows=[...])`, the instances belong to the caller. Rendering writes the injected `input`/`expected_output` keys into `variables`, so doing it in place would make a second run over the same list resolve `{{input}}` against the first run's already-rendered value — a CI retry silently evaluating different data. `_normalize_inline_rows` builds fresh rows with fresh variable maps; keep it that way. The `dict()` copy there is also load-bearing for a second reason: `parse_template` resolves placeholders via `isinstance(value, dict)`, so any other `Mapping` would leave every placeholder literal and send raw mustache text to the model.
+
 ---
 
 ## Adding a New Export