Research Factory: Repeated Research With Held-Out Proof

What Shipped

Research Factory turns one-off runs into a research programme with explicit operating limits and reviewable evidence.

Each Factory contains Efforts: durable objectives that can launch research runs, launch typed maintenance runs, and schedule the next cycle. A research run investigates the objective and produces evidence or a candidate. A maintenance run reads the current state and records what should happen next. Both remain normal Managed Research runs with the same receipts, budgets, and terminal states.

The first release includes:

• Factory budgets and active-run caps;
• durable Efforts with research and maintenance run kinds;
• typed Factory status, experiment history, candidate evidence, and WorkProducts;
• scheduled wakes with a dry-run preview;
• immutable candidate identity for benchmark-owned held-out grading; and
• Python SDK and MCP paths through synth-ai[research] 0.15.0.

The control loop is deliberately evidence-first. A Factory can produce a candidate, but it cannot declare that candidate a champion by reading its own training traces. Promotion requires an evaluator outside the worker's workspace to check out the immutable candidate and grade it on held-out inputs.

The Craftax Acceptance Proof

We used Craftax to prove that lifecycle end to end. The Factory had to operate inside a bounded project and Effort, produce inspectable work, retain an immutable candidate identity, survive benchmark-owned grading, and clean up its rehearsal resources.

The durable receipt spans Factory creation at 13:42:25Z through grading at 14:41:22Z, about 59 minutes. The held-out evaluator reports 0.328s for its five-seed policy sweep. The packet does not record a total run cost, so we do not publish a cost figure for this acceptance run.

These numbers describe different layers of the receipt. The 1.0 reward is the FactoryBench acceptance verdict for the lifecycle. The held-out mean and benchmark score describe the immutable candidate that the benchmark graded. The proof is that the Factory produced something the benchmark could identify, replay, and judge—not that the Factory discovered a new Craftax champion in this run.

A Separate Local-Harness Result

We also have an earlier Craftax result from the non-agent local harness local-harness-20260705T000000Z-always-on-local. That harness measured a +3.7156 mean delta over n=64, with a 95% confidence interval of [3.1859, 4.2422]. Its archived summary does not include total cost or wall time, so the claim is limited to the recorded paired-evaluation result.

That result is a separate evidence tier. It was not produced autonomously by the Research Factory, and we do not combine it with the FactoryBench acceptance run. We include it because it shows the kind of benchmark signal a future Factory can search against, while the acceptance run proves the gated product lifecycle around that search.

Start With One Bounded Effort

Install the Research extra:

pip install "synth-ai[research]==0.15.0"

Then create one Factory and one Effort, link a runnable project, launch a research cycle, and inspect the typed status before scheduling another wake. The Research Factory quickstart shows the Python and MCP paths, including budget policy, maintenance runs, evidence inspection, dry-run scheduling, and rehearsal cleanup.

The full lifecycle lives under SynthClient().research.session. The shorter SynthClient().research.factories namespace remains the Synth Tag surface for one bounded delegated task.

Release Boundary

This release proves a gated research lifecycle and immutable held-out grading. It does not claim a 24/7 reliability window, autonomous uplift from the local harness result, or Factory-to-pull-request code delivery.

Those are separate gates. Shipping the narrow boundary now keeps the receipt honest: objective and budget in, inspectable evidence and independently graded candidates out.

Try It

Follow the Research Factory quickstart, or start Managed Research with one bounded objective you can evaluate on held-out inputs.