LATEST STATE ONLY
Current-only
π2 × A2 → π3
One moving lineage. After π0, π1, and π2 co-evolve with the archive, the next update must use the latest state: π2 × A2 → π3.
Policy-Archive Search at Test Time
PAST.
Reusing trainable states,
not just solutions.
A solution is not the only useful thing a search leaves behind.
PAST gives earlier learned states another opportunity to continue.
THE CORE IDEA
three moments
An earlier state and the current archive feed a new continuation.
The conceptual sequence shows how PAST keeps both kinds of history available without rolling the Solution Archive back.
selected learned statecurrent solution contextnew continuation
01 The idea
Which learned state should
receive the next update?
Test-time training changes both the solutions we have found and the policy that proposes them. Keeping a solution does not preserve the trainable state that produced it.
proposal distribution
sampled candidates
verified Solution Archive
HISTORY REMAINS ELIGIBLE
PAST
πk≤t × At → π3
Cross-time pairing. After the same π0, π1, π2 history, PAST selects π1 as πk and pairs it with the current A2 to produce a different π3 continuation for verification.
STRUCTURAL DIFFERENCE
Current-only keeps one continuation eligible. PAST keeps the latest option and reopens earlier trainable distributions under today's archive, turning training history into a continuation search space.
See the frozen common-context probeSchematic proposal landscape, not a measured embedding. PAST enlarges the set of eligible continuation distributions; verified outcomes, rather than cloud position, determine whether a branch improves the frontier.
02 A real strategy shift
Back to the root.
Forward to a better search plan.
After 21 updates on AC1, PAST revisits the root trainable state under the current Solution Archive. The resulting child does not merely tune a constant: its verified candidate reorganizes the search around a six-strategy portfolio.
Recorded stateSelected parentNew child
Observed candidate rewrite
The continuation changes the search plan.
The prompt shows a reference program that starts from one construction supplied by the current archive. The verified E22 candidate ignores that provided start, builds six qualitatively different initializations, scores every one, then sends the winner into LP-guided refinement.
E22 / verified frontier shift
1.511776188Frontier before E22
1.508246502New frontier, lower is better
−0.003529685 in the recorded AC1 trajectory. The improvement is verifier-observed; the page does not turn this single branch into a causal ablation.
Inspect the raw strategy change
Audit trail
From program structure to verifier result.
The archived input program, exact model response, parsed candidate, and verifier record remain available below. They support an observed rewrite and score transition, not a controlled causal attribution to any one component.
View the exact E22 program- Selected state
- policy-root
- Latest state then
- policy-0021
- New child
- policy-0022
- Current archive
- 588 → 602 retained solutions
03A What is being searched?
Three problem spaces.
One test-time protocol.
PAST is evaluated through the same program-generation and verifier interface on three very different discovery problems. The page keeps the task semantics, model protocol, and comparison horizon visible instead of hiding them behind a single score.
COMMON PROTOCOL50 optimizer updates
Every direct batch generates 8 candidates from each of 8 retained solution parents. A historical revisit adds one fresh on-policy batch; those extra candidates are counted in the matched-rollout view.
The CP26 ProgressEMA S1 lane uses a shared five-update warm-up prefix; AC1 and Erdős start from the policy root. The page reports this as a trajectory description, not as a cross-task seed claim.
03B What PAST preserves
Two archives.
One moving search.
The key intervention is not a new verifier or a larger candidate batch. It is the separation of two kinds of state: trainable policy history and verified solution context.
POLICY ARCHIVE
Recoverable learners
LoRA adapter + matching optimizer state, retained as a lineage of possible continuation parents.
rootearlier stateselected statelatest state
×selectProgressEMA
or control
or control
SOLUTION ARCHIVE
Verified context
Admitted constructions and their evaluator context continue accumulating across the run.
verified solutionverified solutionnew contextfrontier
→continuefresh on-policy
candidates
candidates
NEW CHILD
Another branch
The selected policy learns under the current archive and is appended as a new child.
historical state + current context
pchild = Update(pselected,
SolutionArchivecurrent)Choosing the latest state every time recovers Current-only.
03C Why keep history?
The same state can meet
a different archive.
A frozen common-context probe asks whether different checkpoints behave identically after the Solution Archive has moved forward. It is a motivation diagnostic: it establishes opportunity, not online selector superiority.
01Freeze six policy checkpoints
02Expose all of them to the same final context
03Generate 8 × 8 fresh rollouts per cell
Every checkpoint wins somewhere.
Across eight fixed solution contexts, the hindsight winners include Base, E5, E15, E25, E35 and E45. The ranking is contextual, not a universal “older is better” rule.
Show seed
Winner probability under the common final contextseed A
rarely winsoften wins
03D What the selector actually did
Allocation is part of
the result.
Three PAST selectors all access policy history, but they create different temporal search patterns. These are descriptive summaries of one completed trajectory per task, not causal rankings.
| Task | Selector | Historical choices | Distinct parents | Mean age | Frontier events | Events / 1k rollouts |
|---|
Not every archive is the same tree.
Compare longest path and leaves to see how selection changes the genealogy, even when every run ends with 51 policy nodes.
Where improvements appeared.
The attribution is chronological: it describes observed batches and does not assign causal credit to a selector decision.
A “meaningful event” means an oriented frontier increment of at least 10−6. Historical-fresh batches depend on solution and policy archives created earlier in the same trajectory.
03E Why restore the optimizer?
The state is more than
the adapter weights.
At one controlled CP26 branch point, matching optimizer restoration changed local proposal quality. It did not, by itself, produce a better retained frontier. Both facts belong on the page.
mean reward
+0.3279
validity
+0.1250
candidate best
+5.69e−6
matching optimizerfresh optimizer
Scope64 shared-seed positions
Timeouts0 matching / 1 fresh
Retained frontiernumerical tie
Interpretationlocal proposal quality, not final superiority
ARCHIVE FOOTPRINT2.537 GB
Approximate adapter + optimizer state per recoverable policy state.
50 STATES126.841 GB
Linear logical storage before the shared base model.
OBSERVED I/O SHARE0.048–0.062%
Persistence and optimizer restore components across the three trajectories.
The I/O share covers measured persistence and optimizer-shard loading components; it is not a full wall-clock or FLOPs accounting. The archive trade-off is part of the method, not a hidden implementation detail.
03 The method
Keep the state.
Keep the new solutions.
A Policy Archive stores LoRA adapters with their matching optimizer states. A separate Solution Archive retains verified constructions discovered throughout the run.
-
01
Evaluate
Evaluate the newly trained policy and admit useful verified solutions.
-
02
Select
Allocate the next update to an eligible archived state. ProgressEMA uses observed capability and delayed progress.
-
03
Continue
Restore the selected trainable state. A historical revisit collects fresh on-policy candidates under the current archive.
-
04
Append
Train the restored state and add its child as a new continuation branch. Evaluate that child in the next round.
Choosing the latest eligible state every time recovers Current-only's single-chain continuation.
04 Discovery results
Three tasks.
Two budget alignments.
Historical revisits generate additional fresh candidates. An equal training horizon and an equal generation budget answer different questions.
| Method | Best found | Epochs | Actual rollouts |
|---|
Main trajectories, not cross-seed means. No task-level error bars are shown here: each curve is one completed trajectory per method and task. The A/B/C controls in the frozen opportunity probe are shared- context probe seeds, not independent training replications. Matched rollouts align candidate generation, not training FLOPs or wall-clock time. UG-TTT retains its five-member ensemble cost.
Across all tasks
At 50 training epochs
ProgressEMA has a better best-found frontier than both baselines on all three tasks.
At matched actual rollouts
CP26 reaches a numerical tie. ProgressEMA retains a better frontier on AC1 and Erdős.
Continuation allocation
Historical-state access alone does not determine the result. ProgressEMA is best or tied-best among the three evaluated PAST selectors at 50 epochs.
04A Full selector comparison
Same search interface.
Different allocation rules.
The headline chart above isolates the paper's core comparison. This table exposes the complete five-method panel at the common 50-update horizon, including the two PAST controls that are easy to miss in a three-line plot.
| Task | Direction | Method | Endpoint | Actual rollouts | Meaningful events |
|---|
Current-only has no policy archive. PAST / Uniform, ScorePUCT, and ProgressEMA share the same continuation machinery and differ in how they choose a parent.
Program generation, verifier re-execution, 8 solution parents, 8 candidates per parent, and the 50-update outer-loop horizon are held in the common protocol.
They are one completed trajectory per method and task. This page does not turn them into significance, compute-efficiency, or causal claims.
05 Round atlas
Every round, not just the headline.
Inspect the complete 50-round control trace without loading the multi-megabyte epoch payloads. Each row keeps the distinction between direct evaluation, historical fresh evaluation, selector choice, archive changes, and frontier events.
E8 / meaningful frontier
Round detail
Counts are persisted archive summaries. “Fresh” means additional on-policy candidates from a historical revisit; a direct frontier event is not attributed backward to the selector decision that follows it.
06 Behind the result
A branch you can inspect.
The selected cases retain their exact generated program, prompt, response, and verifier record. No model text has been rewritten for the presentation.
Sources and scope
Comparison curves come from the frozen, cross-checked rollout artifacts used for the paper's matched-budget analysis. The branch case comes from the published CP26 trace. Local source hashes are retained in the downloadable provenance record.
The complete trace includes admitted solutions, including those pruned later. Invalid and never-admitted rollouts, optimizer tensors, adapter weights, and machine metadata are excluded. Existing Python environment-prefix redactions in public prompt and response text are preserved.
Frozen-policy opportunity is not proof of online selector superiority. Optimizer restoration shows a local proposal-quality benefit in the controlled CP26 intervention, not a better final frontier in the available comparisons.