Scaling Train Time Compute for Gepa
Scaling train-time compute for GEPA across public task containers — same-container comparisons, coverage curves, and proposer scaling evidence.
The prompt-optimization loop is simple to describe: propose a candidate, run it against a task, score the result, and use the evidence to propose again. The hard part is keeping the task boundary stable while teams change languages, scoring code, datasets, and model providers underneath it. Without that boundary, every optimizer integration becomes a custom harness, and every result is harder to reproduce.
Synth Containers make that boundary an HTTP contract. The optimizer does not import a task package, read local dataset files, or learn how a harness happens to be implemented. It calls a small set of routes, receives rows and rewards, and treats the container as the source of truth for the task. The task can be a classifier, a coding agent, a QA system, or a long-horizon environment; the optimizer sees the same surface.
GEPA is the first optimizer we are shipping on top of this interface. We started with public cookbook tasks because the contract needs to be inspectable: the container code is public, the chart rows are generated from recorded run artifacts, and publication requires an exact evidence commit for the generated chart rows.
In this post, we describe the container contract and report the same-container comparison against gepa-ai on the current four-container comparison set: HealthBench Pro, tau2-bench retail, Banking77, and HotpotQA. A proposer-model run group on HealthBench Pro and tau2-bench retail is reported in the Proposer Scaling section as a sanity check, not as a full scaling law.
Policy Scaling
Thesis: Same-container comparisons are useful only when the task boundary, policy model, split shape, and evaluator are held constant.
The initial-post scope is four containers in the current comparison set: HealthBench Pro, tau2-bench retail, Banking77, and HotpotQA. Rows with evidence were run through the same container boundary for Synth GEPA and gepa-ai, then re-scored posthoc on the same heldout seeds. Train split metadata is shown only where the producer summary exposes a train denominator. Candidate counts, rollout calls, and wall-clock time vary by run, so this is same-container evidence rather than an equal-compute benchmark.
Same-container head-to-head
Synth GEPA vs gepa-ai on tau2-bench retail, HealthBench Pro, HotpotQA, and Banking77. Each row uses the same container boundary, policy model, train split, and heldout split.
| Task | Bestheldout | Heldoutpareto K | Trainpareto K | Jointpareto |
|---|---|---|---|---|
tau2-bench retail heldout → Synth | gepa0.400Synth0.430 +0.030 | gepa62/100Synth63/100 +1 | gepa22/30Synth26/30 +4 | gepa1 · 50%Synth1 · 50% 0 |
HealthBench Pro heldout → Synth | gepa0.353Synth0.361 +0.008 | gepa157/200Synth158/200 +1 | gepa86/100Synth77/100 -9 | gepa3 · 50%Synth3 · 50% 0 |
HotpotQA heldout → gepa-ai | gepa0.748Synth0.707 -0.042 | gepa145/200Synth142/200 -3 | gepa—Synth— — | gepa5 · 83%Synth1 · 17% -4 |
Banking77 heldout tie | gepa0.785Synth0.785 tie | gepa182/200Synth177/200 -5 | gepa100/100Synth97/100 -3 | gepa3 · 60%Synth2 · 40% -1 |
Read: best heldout is the strongest single candidate; heldout/train pareto are cumulative rows solved by any candidate through K; joint pareto is the 3-objective frontier (heldout, cost, time).
evals/evidence/benchmarks/*/summary.json.Coverage tells a different part of the story than best-candidate score. On HealthBench Pro, Synth and gepa-ai are close on cumulative heldout coverage (158/200 vs 157/200) even though Synth has the best heldout candidate in this final run. On tau2 retail, Synth is slightly ahead on both coverage (63/100 vs 62/100) and best heldout (0.430 vs 0.400) under the fresh same-container rerun. Banking77 ties on best heldout while gepa-ai covers more rows (182/200 vs 177/200); HotpotQA goes the other way, with gepa-ai ahead on both best heldout (0.748 vs 0.707) and coverage (145/200 vs 142/200).
Proposer Scaling
Thesis: Changing proposer compute should help most when the task gives the optimizer expensive, high-signal rollout feedback.
The launch proposer sweep is deliberately small: HealthBench Pro and tau2-bench retail, each with gpt-5.4-nano, gpt-5.4-mini, and gpt-5.4 proposers. Those are the proposer labels recorded in the run manifests. These Chart D runs skipped heldout scoring by design, so the chart reports observed optimization reward from each Synth GEPA run rather than posthoc heldout.
What Changes
Thesis: GEPA changes candidate fields, not the container contract. The container still owns rows, scoring, rollouts, traces, and task secrets.
The mutable field changes by task, but the optimizer contract does not:
- HealthBench Pro:
stage1_system - tau2-bench retail:
domain_policy - Banking77:
stage2_system - HotpotQA:
stage1_system
Those fields are the only task-specific program surfaces the optimizer mutates.
Open Sourcing GEPA
Thesis: Open sourcing GEPA is not just releasing an optimizer. It is releasing the task boundary, evidence format, and runtime interfaces that make optimizer results inspectable.
The systems story is that the environment around the optimizer is part of the product. For GEPA, that environment is the container contract and the durable optimizer runtime. The optimizer should not know whether a task is a Python classifier, a Rust program, a TypeScript service, a game environment, or a medical-rubric evaluator. It should see rows, candidates, rollouts, rewards, traces, and usage through one stable surface.
The diagrams below are organized as subsystem views rather than one monolithic architecture figure. Start with the container boundary, then inspect the GEPA runtime, then the Flash Evolve scheduling layer.
Container boundary
The task stays behind a small HTTP contract. Rows, rewards, scoring, traces, and secrets remain container-owned.
Implementation notes+
The contract is deliberately small. A container tells the optimizer what mutable modules exist, which dataset splits are available, how to fetch rows, and how to score a candidate on a row. Everything else stays behind the container boundary.
- A HealthBench rubric task, a tau2 retail workflow, a Banking77 classifier, and a HotpotQA question-answering run should not require four optimizer integrations.
- They should require four containers that implement the same routes.
| Route | Method | Optimizer use |
|---|---|---|
| /metadata | GET | Confirms GEPA contract version before the run starts |
| /program | GET | Reads the mutable modules and seed candidate |
| /taskset | GET | Gets the taskset metadata advertised by the container |
| /taskset/tasks | POST | Fetches train, minibatch, reflection, and heldout rows |
| /rollout | POST | Submits a candidate and row, gets back reward and usage |
| /task_info | GET | Optional task context for proposer prompts |
Resources
- synth-cookbooks-public
- GEPA launch evidence packet
- Prompt Optimization Overview
- gepa-ai reference implementation