Experiment

Expand sweep matrix / single-run / multi-run to a flat list of primaries.

Each entry is a "group" when n_repeats > 1. See :meth:_replicates_for for the per-primary seeded replicates.

Per-primary seed replicates.

For n_repeats == 1: returns [(primary, None)] - no group. For n_repeats > 1: returns N tuples of (\<RunConfig with name=primary.name__s\<seed> and seed=\<seed>>, primary.name). Seeds run [base_seed, base_seed+1, ...] when base_seed is set, else [primary.seed, primary.seed+1, ...].

Normalize metadata.benchmark to a list of (Benchmark, spec).

Accepts three shapes:

• None / empty → [] (no eval).
• single dict (legacy single-benchmark form) → wrapped into a one-element list with name defaulting to "benchmark" (or the spec's own name if present).
• list[dict] (new multi-benchmark form) → each entry must carry a name and may carry tags and run_every.

self._benchmark_override (test seam) bypasses everything and returns a single-entry list.

Push the arm's local metrics.jsonl rows to the store.

Runner-time logging writes locally; this batch-forwards to the dashboard so the script doesn't have to call backfill_step_metrics manually after training.

Reconstruct the run output dir the runner wrote into.

User-supplied factory wins; otherwise pick a default by backend kind.

Falls back to the registry's get_default_inference_factory so we don't have to import backend-specific inference modules here - e.g. tinker registers its own default at module load.

Score each post-training benchmark and attach an EvalResult per entry.

score_api_models=False skips the closed/API-model (benchmark.models) evals - used for continual stages after the first, since a closed model's weights are fixed across stages so one scoring suffices (only the trained checkpoint, which changes per stage, is re-scored).

Entries flagged with run_every are in-loop and skipped here (their scoring happens during training in the algorithm wrapper - task commit 2). Entries without run_every get a single post-training EvalResult appended to arm.evals.

After all benchmarks score, the flat back-compat fields (arm.eval_metrics / eval_breakdowns / eval_seconds) mirror the primary eval - the first test-tagged post-training row, else the first post-training row, else nothing.

Score one benchmark through harbor's rollout engine and upload the eval rollouts (kind='eval'). Opt-in via benchmark.engine: harbor.

api_model (a litellm string) scores a closed / API model instead of the trained checkpoint - same rollout path, model_client='litellm', recorded as its own per-model eval. None → the trained checkpoint.

The trained sampler checkpoint URI from the arm's artifacts.

Train the base run once per dataset in continual.datasets, in order, chaining each stage's final weights (fresh optimizer) into the next. Each completed stage is scored on every benchmark via :meth:_execute_arm; a chain stops at the first stage that does not complete.

With n_repeats > 1 the whole chain is replicated once per seed ([base_seed, base_seed+1, ...] or [run.seed, ...]). Stage i across all repeats shares one dashboard group, so variance is aggregated per stage. Within a chain, weights are chained only between that chain's own stages.

The full training-state path (weights + optimizer) of an arm's final checkpoint. Continual learning loads weights only from this into the next stage. Distinct from :meth:_final_checkpoint, which returns the inference-only sampler path.

Register a run group for variance studies; returns its id (or None).