GELO: Archive-Based Optimization for Long-Horizon Agents

The Algorithm

Prompt optimization becomes qualitatively harder when the task is an episode rather than an answer. On a static benchmark, a candidate prompt fails at the final prediction. In an agent environment, the same candidate can be locally good and globally unusable: it may reach the furnace and then loop, collect iron and craft the wrong item, or spend the evaluation budget redoing an opening route after the world state has changed.

This creates a credit-assignment problem that ordinary prompt search is not designed to solve. A full-episode score entangles three distinct questions: whether the policy can return to the right region of state space, whether it can execute the next skill from that region, and whether the resulting behavior survives heldout seeds. GELO separates these questions by making the checkpoint, the local objective, and the promotion rule explicit.

The design is inherited from Go-Explore, which was introduced for hard-exploration reinforcement-learning problems. The central move is simple: maintain an archive of promising states, return to one before exploring, and use the resulting trajectories to expand the archive. The later Nature version, First return, then explore, framed two common exploration failures as detachment — forgetting how to reach useful states — and derailment — exploring before returning to the intended state.

GELO ports that idea into long-horizon language-agent optimization. Atari cells become hosted checkpoints. Game-score deltas become verifier evidence, achieved labels, reward vectors, and heldout replay. Random exploration becomes candidate generation by proposer models, local hill-climbs, and archive-level consolidation. The object of search is no longer only the prompt; it is the tuple:

(checkpoint region, local objective, candidate family, evidence, promotion rule)

At the top level, GELO is an archive loop rather than a single mutation loop:

1. Archive. Store checkpoints, trajectory segments, policy candidate IDs, achieved labels, heldout evidence, and lineage.
2. Select themes. Identify frontier regions where the agent is close to a meaningful new achievement or repeatedly fails in a consistent way.
3. Hill-climb locally. Run scoped candidate searches from each theme checkpoint, with a local success metric and a bounded rollout budget.
4. Consolidate. Re-score winners on heldout seeds; promote only candidates that improve the archive or expand coverage.
5. Replay and relabel. Use heldout replay to update evidence, label new achievements, and expose the next frontier.

1archive = seed(baseline, fresh_rollouts)   # candidates, checkpoints, heldout evidence
3while budget_remains:
4    themes = select_themes(archive)        # checkpoint region -> next objective
5    for theme in themes:               # local hill climbs, run in parallel
6        checkpoint = theme.checkpoint  # remember, return
7        slate = propose(theme, archive.evidence, plugin)
8            # plugin: prompt edits · BC beta · future RLVR / OPSD
9        for cand in slate:            # ...then explore from it
10            rollouts   = resume(checkpoint, cand)   # branch from checkpoint
11            cand.score = local_objective(rollouts)
12        theme.champion = argmax(slate)
13    archive = consolidate(archive, champions)  # promote winners, retire themes
14    relabel_heldout(archive)                 # fresh seeds, achievement labels
16return archive.best_on_heldout           # plus the frontier map

A theme is a typed local hypothesis: from this checkpoint distribution, optimize this next objective under this success metric. Themes prevent the optimizer from collapsing into one global prompt search. They also make the evidence auditable: a candidate is not merely “better”; it is better for a named frontier such as preserving the opening route, converting stone into a stone pickaxe, recovering coal after iron, or descending after locating a staircase.

The candidate surface is intentionally broader than a single system prompt. A candidate can change policy instructions, strategy context, tool-use rules, verifier-facing guidance, termination criteria, and attached skills. The archive tracks which candidate families earned evidence and which themes they cover.

This is the main algorithmic distinction from prior prompt optimizers. MIPRO and GEPA decide how to update a prompt or language-program component after observing evaluation evidence. GEPA, in particular, uses natural-language reflection over trajectories and a Pareto frontier to combine complementary lessons. GELO keeps that style of reflective candidate generation, but adds the long-horizon decision that GEPA does not try to solve directly: which checkpoint should generate the next data? The difference is not “better prompt search”; it is prompt search conditioned on an archive of partial progress.

We are not using this post to claim a completed causal ablation of checkpoints versus fresh-start search. The paired checkpoint-on/off and warm-archive/fresh-start ablation bar was not met, so the mechanism stays framed as the optimizer design and as observed search behavior, not as an isolated causal effect size.

One Run, End to End

The Crafter furnace subgoal run is a compact example: 22 minutes, three proposer rounds, three themes, eleven candidates, and heldout furnace placement moving from 0% to 17%.

The graph should be read as a search lineage. Time runs left to right. The baseline seed fans out into theme hill-climbs; a theme champion fans in at consolidation; refinement candidates fan back out from the promoted state; surviving branches fold into the archive. Gray edges mark candidates that were useful as evidence but did not remain active after consolidation. This is the unit of analysis GELO is meant to expose: not just the best final prompt, but the route by which a local frontier became an archive item.

Craftax

The strongest evidence so far comes from Craftax, a JAX-based benchmark designed to combine the speed of vectorized RL environments with the long dependencies of open-ended worlds. Craftax-Classic reimplements Crafter in JAX; the full Craftax environment extends the mechanics with richer dependencies and NetHack-like elements.

The first GELO pass produced a real but narrow improvement: the archive learned a route from early resource collection into basic tool use. That result was useful, but it was still an opening-route result. We therefore replayed the prior checkpoint with a higher-throughput proposer contract: maintain at least eight active themes, and explicitly allocate theme families to route preservation, stone-pickaxe conversion, coal and iron harvesting, furnace placement, iron-pickaxe conversion, diamond navigation and mining, and deep-run survival.

Two successful proposer iterations expanded the archive to 14 themes and 34 candidates, 27 of them new. Given the available budget and the current archive state, the search diversified instead of only polishing the opening route.

What Improved

The strongest completed heldout evidence is now midgame behavior. The four candidate families in the trace are all prompt-optimization artifacts — no supervised finetuning or policy-gradient update was used:

The qualitative shift matters. The earlier route-following candidates showed that the policy could collect wood, place a table, make a wood pickaxe, and collect stone. The higher-throughput run pushed heldout trajectories into coal, stone-pickaxe, iron, and furnace states. “The agent sometimes reaches stone” and “the agent reaches iron and furnace, then fails to make the correct next tool” are different research problems. The latter is narrower, easier to instrument, and better shaped for the next GELO round.

What Did Not Work Yet

We did not collect diamond. Across the focused deep probes and targeted manual probes, 80 completed heldout rollouts produced zero diamond labels.

We also attempted the longer-horizon Craftax refresh that would have upgraded this ceiling readout. It did not produce a terminal run: every viable Cx1 launch path eventually hit a container final-record boundary where a completed rollout returned 404 from /rollouts/<id> before the runner could read a usable final record. That failed packet is retained as evidence for the fallback, not counted as a diamond result.

The most informative trace was also the clearest failure. One rollout reached make_stone_pickaxe, collect_iron, and place_furnace, then spent dozens of turns without further progress. In one near miss, the policy attempted make_iron_sword instead of entering the iron-pickaxe branch. A manual follow-up prompt removed the sword mistake and reached an 8.1 reward spike, but it still missed coal and never completed make_iron_pickaxe.

That trace defines the next frontier. The next themes should target three checkpoint-local failures: avoid looping after furnace placement, fetch coal when iron is already in inventory, and force the iron-pickaxe craft when the recipe is nearly satisfied rather than drifting back into exploration. This is the intended shape of the algorithm: each archive improvement should turn a vague long-horizon failure into a small set of named subproblems.

NetHack

To test whether the loop transfers outside crafting environments, we ran GELO on NetHack through the NetHack Learning Environment (NLE). NLE is a scalable, procedurally generated, stochastic roguelike benchmark for RL research, with long-horizon demands on exploration, planning, skill acquisition, and language-conditioned behavior.

We optimized a deliberately simple metric: scout score, the number of distinct (dungeon level, x, y) tiles the hero occupies in an episode. Scout score is not a complete measure of NetHack competence, but it is hard to game by local looping: standing still or walking back and forth over known cells adds no new points.

The protocol held the environment interface fixed. We used a gemini-3.1-flash-lite policy and a gpt-5.4-mini proposer, evaluated on fixed heldout seeds, 200-step episodes, with structured observations fixed and only the policy prompt changing. Candidate proposals were grounded in NetHack-specific behavior — corridor-first exploration, quadrant sweeps, frontier navigation, glyph awareness, and staircase reasoning — rather than leaking Craftax recipes into a new domain.

On a single heldout seed, the occupancy maps below show every tile the hero stood on during the episode, before and after optimization, in the same coordinate frame:

Coverage also increased with a longer episode budget. At a 350-step budget, the same policy reached a 48.8 mean scout score, with the best seeds exceeding 70–80 distinct tiles.

The intervention is a behavioral reallocation, not a uniform improvement. Exploration and discovery achievements rise, while combat, inventory, and score milestones fall. That is the correct tradeoff for this objective: the optimizer pays for new ground covered, not for fighting or looting.

The main limitation is still structural. Almost all coverage remains on dungeon level 1. Descending requires finding a > staircase and executing the transition reliably; prompts that simply rush for stairs scored below baseline in these budgets. The two interventions that did move the metric were structured context — position, visited cells, blocked directions — and survival. Early death remains the largest source of lost coverage. Multi-level descent is the next NetHack frontier, in the same way that diamond is the next Craftax frontier.

Crafter

Our first Crafter evaluation used mean reward. We replaced it because the aggregate was too coarse to support a strong claim. A mean-reward curve can improve while hiding which rung of the crafting tree moved, and it can punish useful local progress that does not survive to the end of the episode.

The rebuilt evaluation uses five binary subgoal sub-environments along the crafting tech tree: wood pickaxe, stone pickaxe, furnace, iron pickaxe, and diamond. Each environment asks one question: did the agent unlock the target achievement? GELO runs separately on each subgoal, in parallel, with reward shaping aligned to that target.

The baseline reliably reaches the first rung — a wood pickaxe on 70% of heldout seeds — and then stops: 0% stone pickaxes, furnaces, iron pickaxes, and diamonds. That clean wall is what makes the subgoal framing useful.

Per-subgoal optimization produces specialists. The furnace champion is the only policy that places a furnace, moving the heldout rate from 0% to 17%; the wood-pickaxe champion moves the first rung from 70% to 83%.

The deeper result is a negative result. Increasing the deeper sub-environments to three proposer rounds, 10–12 candidates, and three themes each did not solve the underlying policy limitation. Stone-pickaxe achievement moved only to 3%; iron pickaxe and diamond remained at 0%. Prompt search can improve the behavior the policy is capable of expressing, but it cannot reliably induce a weak policy to execute a precise “stand at the table holding stone and craft” sequence when the base model does not have that competence. GELO exposes that ceiling rather than hiding it behind an average reward.

A Compute-Scaling Probe

After the subgoal guardrail, we ran three independent Crafter GELO jobs in parallel using DeepSeek Flash v4 for the proposer, data-miner, verifier, and terminator lanes. This is not a scaling law. The narrower hypothesis was operational: can the same inexpensive-model configuration run long enough, across three independent jobs, to produce archive promotions and heldout movement?

All three jobs completed, none crashed, and every job promoted at least two candidates.

The run-to-run spread is informative. One run kept only two themes but produced four promotions; another kept six themes and produced the strongest theme lift. In this development probe, the archive histories fan out in different places while the heldout readout moves in the same direction.

The caveat is equally important: the logs include skipped data-miner and consolidation substeps caused by JSON/schema failures. This is a successful development probe, not evidence that the scaling stack is solved.

Exploring Posttraining Data

GELO is useful not only because it finds better prompts, but because it changes the shape of the data we can train on.

A conventional rollout dataset is dominated by prefixes. In a long-horizon environment, many examples show the policy collecting wood, walking corridors, or surviving the opening because those states are easy to revisit from reset. The rare states we care about — holding iron next to a furnace, standing near a staircase, recovering after a route deviation — appear sparsely, if at all. Training on that distribution gives the model many copies of the opening and very little pressure on the frontier.

GELO produces a different dataset: frontier-conditioned experience. For each archived checkpoint, we can collect positive and negative attempts against the same local objective; label whether the policy returned to the checkpoint distribution; separate route failures from skill failures; and replay promoted candidates on heldout seeds. This turns a raw episode trace into supervised and reinforcement-learning objects with cleaner credit assignment.

Concretely, the archive yields several posttraining views:

• Return data: trajectories that recover the archived state from earlier checkpoints.
• Local skill data: attempts from the checkpoint to the next achievement, including near misses.
• Contrastive failures: candidate pairs that share the same prefix but differ at the frontier.
• Verifier data: success and failure labels tied to explicit achievements, not just scalar episode reward.
• Curriculum data: a lineage of promotions that orders frontier skills by when the current policy could make use of them.

This is why we view GELO as an optimizer and a data engine. Prompt optimization is the first consumer because it gives immediate feedback and fast iteration. The beta behavioral-cloning lane can use promoted traces and corrected near misses. Future RLVR and policy-gradient variants can train against scoped objectives rather than full-episode sparse rewards. On-policy self-distillation can use the archive to sample states where the current policy is plausibly one or two decisions away from a new capability.

The important constraint is heldout replay. A checkpoint-local improvement is only useful if it survives beyond the exact seed and trace that discovered it. GELO therefore treats consolidation as part of the algorithm, not as a posthoc charting step.

For the June 26 hosted relaunch, we also ran the new public GameBench Rust smoke presets end to end. These are integration receipts, not new benchmark rows: the Craftax smoke run completed with integrity passing, 12 total rollouts, 108 training checkpoints, heldout mean 1.0, and no promotion because the seed prompt was already the champion; the Rogue smoke run completed with integrity passing, 12 total rollouts, 108 training checkpoints, heldout mean 0.0, and no promotion. The useful result is operational: both presets now exercise the Rust gold service, tunneled ReAct-style task container, prompt-only candidate contract, isolated heldout replay, and terminal GELO artifact packet.

pip install --upgrade --pre synth-optimizers==0.2.6.dev20260626
export SYNTH_API_KEY=...
synth-optimizers gelo startup

# Public container URL
synth-optimizers gelo submit \
  --preset crafter_smoke \
  --container-url https://YOUR-CONTAINER.example.com \
  --follow

# Local container through Synth Tunnel
synth-optimizers gelo submit \
  --preset craftax_gamebench_rust_smoke \
  --tunnel-url http://127.0.0.1:8943 \
  --tunnel-provider synth_tunnel \
  --follow

synth-optimizers gelo watch goex_... --slice board --goex-events

Plugin lanes (invite-only preview)

Hosted GELO's typed plugin graph supports SFT as an invite-only beta lane for allowlisted organizations. We are running private evaluations on RLVR and OPSD treatment variants, and will expand access from the waitlist before making broader availability claims.

Request access through the hosted optimizer docs and mention "GELO plugin lanes" in the request.

Resources

• Scaling Train Time Compute for Gepa — Synth's prior optimizer release and implementation context.
• GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning — reflective prompt evolution with Pareto-based candidate selection.
• Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs — MIPRO / MIPROv2 background for prompt optimization in LM programs.
• Reinforced Self-Training (ReST) for Language Modeling — offline self-training baseline for language-model policy improvement.
• Go-Explore: a New Approach for Hard-Exploration Problems — original Go-Explore preprint.
• First return, then explore — Nature version of Go-Explore and the detachment / derailment framing.
• The Craftax Benchmark and Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning — Craftax and Craftax-Classic.
• The NetHack Learning Environment — NLE benchmark.
• Benchmarking the Spectrum of Agent Capabilities — Crafter benchmark.
• The Swarm — Go-Explore-inspired swarming in PokerL / Pokemon RL.
• Agent Learning via Early Experience — training agents from their own interaction data rather than only expert demonstrations or scalar reward.
• Continual Harness: Online Adaptation for Self-Improving Foundation Agents — reset-free harness refinement during continuous play.
• GELO skill — public runbook for authoring GELO-compatible task containers and preparing hosted optimizer submissions.
• Crafter + GELO cookbook — public walkthrough for a SynthTunnel-compatible hosted GELO task target.
• Evidence provenance — per-chart source identifiers and SHA256 hashes are embedded in the figure footers; raw packet artifacts are retained in Synth's internal launch evidence bundle.

Citation

If you use GELO, the archive-based optimization algorithm, or the concept in academic work, please cite:

@misc{purtell2026gelo,
  title = {GELO: Archive-Based Optimization for Long-Horizon Agents},
  author = {Purtell, Josh},
  year = {2026},
  howpublished = {\url{https://www.usesynth.ai/research/go-explore}},
  note = {Synth Research}
}