ForgeRL / Agent research
Inspect the decision.
Check the result.
Coding-agent experiments with the source, patches and test results kept together. Start with a real recorded run.
Last benchmark run: —
Recorded evidence
Browsing and replay spend no API credits.
A quick look at the system
Follow a real repair, then inspect a case where verification missed a failure.
Later study · September 25, 2026: Four-seed replication & VERIFY ablation — 1,296 attempts ↗. This report revisits the exposed catalog; the explorer below shows the separate transfer pilot or v0.2 study.
- Catalog
- —
- Recorded episodes
- —
- Models evaluated
- —
- Benchmark version
- —
Cost / quality
Success versus accounted cost
Loading recorded measurements…
Policy comparison
Five policies. One protocol.
Missing measurements remain blank.
| Policy | Episodes | Success | Hidden tests | Cost / task | Tokens / task | Latency | Calls | Attempts | Regressions | After repair | Escalations | VERIFY / task |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Loading recorded measurements… | ||||||||||||
Mean metrics are per recorded episode. Hidden-test pass rate and task success are different measures. “After repair” is conditional on repair attempts.
Reproducible tasks
Explore the benchmark
Task brief
Select an authored task
Source files and the success criterion will appear here.
Success criterion
—
No source loaded.
Source is supplied as context to the model; this is not a claim of autonomous file discovery.
Visible checks —
- No checks loaded.
Stored execution
Inspect a recorded trajectory
Select a task to see its available experiment runs.
- Hidden tests
- —
- Accounted cost
- —
- Tokens
- —
- Latency
- —
Observable prompts, supplied files, orchestration, patches and outcomes. Private model reasoning is not recorded.
Playback only. No new model request.
- No recorded run selected.
No recorded patch selected.
No test evaluation selected.
No recorded candidate selected.
Observed failure signals
No recorded run selected.
Labels are observable proxies, not causal explanations of a model’s reasoning. Unmeasured localization, planning or context loss remain unclassified.
Optional fresh trial
Try one new candidate.
One published repair task, one new model request, then isolated test checks. This trial does not change the benchmark.
Recorded replay remains available.
Inspect the fixed taskLimits and privacy
One trial per client network each UTC day. The shared allowance is at most $0.05 per UTC day and $1 per UTC month, within the original project budget. No custom prompts, uploaded code or account registration. Session cookies and a privately salted network identifier enforce admission limits. Model credentials stay in the private worker.
A fresh trial starts only when you press the button.
Inspect this live trial’s recorded patch
Method & boundaries
A routing experiment, kept inspectable.
What the controller chooses
The policy can retry, repair, escalate, roll back or stop. The language models remain unchanged. A learned controller uses training transitions; unseen states use a declared fallback.
What the tasks establish
Authored miniature Python repositories test specific coding requirements. Multi-requirement “long-horizon” tasks are a stress category, not evidence of autonomous work on large production repositories.
What remains hidden
Only visible checks guide a repair. Hidden checks grade the final candidate after routing ends. Their inputs and reference solutions are excluded from this explorer.