Vendor-published Benchmark
Legacy-Bench: frontier agents pass only 16.9–42.5% of COBOL, Java 7 and Fortran tasks
Hundreds of tasks across six legacy language families (46% COBOL, 32% Java 7) with hidden verification tests. Bug fixing scores roughly twice implementation, which scores roughly twice migration. In 97% of failures the agent believed it had solved the task.
The numbers (as reported)
| Pass rate range (12 model–agent combos) | 16.9%–42.5% |
|---|---|
| Same models on Terminal-Bench 2 / SWE-bench Verified | >70% |
| Best overall (GPT-5.3-Codex) | 42.5% |
| Tasks no model solves | 44 (31 of them COBOL) |
| Failures where agent believed it succeeded | 97% |
Facts
- Company / subject
- Factory
- Tools
- Droid, Codex, Gemini CLI
- Codebase
- brownfield
- Work type
- Benchmark
- Stack
- COBOL, Java 7, BASIC, C89, Fortran, Assembly
- Published
- 2026-04-01
- Last checked
- 2026-10-17
Caveats
Published by Factory, whose own agent (Droid) is reported as outperforming providers' agents. Only 10 sample tasks are public. Models evaluated include GPT-5.3-Codex, Gemini-3.1-Pro, Opus 4.6 and GLM-5.