How to Use AI to Modernize COBOL Code: What OpenAI's New Tests Show

You will learn: what OpenAI tested, what the reported scores mean, how to set a practical acceptance test, and which production dependencies remain outside the evaluations.

Two evaluations, not one modernization score

OpenAI reported progress across COBOL, Fortran and Visual Basic, languages with less public training material than Python or JavaScript. It used two different evaluations. The first asks whether a model can build new working software in a specified language. The second asks whether a model can migrate an existing COBOL program while preserving what the program does. Combining them into a single “COBOL modernization rate” would misrepresent both.

OpenAI has published the MultiLangBench setup and test suite. Its README says the evaluator uses the reference SQLite implementation to check outputs, and reports test pass rate alongside API cost. The COBOL migration evaluation is described in OpenAI's release as using hidden tests; the release does not say those hidden cases are public. A team cannot reproduce its migration score merely by downloading MultiLangBench.

What the October 19 results actually say

On MultiLangBench, GPT-6 Astra averaged an 87.4% test pass rate across 20 languages at maximum reasoning effort; GPT-5.6 Sol's reported average was 26.8%. OpenAI reports gains in every tested language, including COBOL from 10.9% to 72.1%, Fortran from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%. Those are scores for building a SQLite-compatible implementation, not for translating a bank's existing application.

On OpenAI's separate COBOL migration evaluation, GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same setting. OpenAI says the newer model's estimated API cost was 39% lower in that comparison. At higher reasoning effort, the reported accuracy reached 97.9%. The test asks for byte-for-byte parity in observable outputs. OpenAI's own example is fixed-point arithmetic: rounding a value when the COBOL program would discard extra decimal digits can produce valid-looking code with a wrong financial result.

These are OpenAI's evaluations and settings, checked on 21 October 2026. They are strong evidence of progress on the specified tasks, not a measured result for your company or a guarantee that the cheapest reasoning setting is right for every case.

A practical way to evaluate one COBOL migration

Treat the model as a candidate translator and analyst. Keep the acceptance decision with the engineers who know the source system.

1. Choose a bounded program and a reason to change it. Record its callers, files, input formats, outputs and business owner. IBM's modernization overview points out that the work can include integration, data architecture, transaction integrity and security, beyond a language rewrite.
2. Capture the current behavior before translation. Create examples for ordinary cases, missing records, boundary values and fixed-point arithmetic. Preserve the source program's observed output as the comparison target.
3. Ask for a candidate and an explanation of uncertain rules. Give the model the relevant code, copybooks and interface contract. Require it to name ambiguities instead of silently filling them in.
4. Run old and new paths on the same cases. Compare outputs exactly where exact parity is required. Log each mismatch with the input, old output, new output and the rule owner who will decide whether it is a bug or an intentional change.
5. Review the surrounding system before cutover. A passing code-level parity suite does not settle batch timing, I/O, job control, access control, data conversion or rollback. Microsoft's COBOL migration account describes these nonfunctional dependencies as a reason some modules need redesign rather than direct translation.

This sequence is a recommended evaluation method drawn from the source material, not a test Xtrusio has run on a customer's codebase. The same review discipline matters in smaller projects: our guides to checking Claude-written Python and QA automation with Claude Code show how to inspect generated code and test outcomes. For a COBOL procurement trial, ask each vendor to show the old behavior, the candidate behavior and the exceptions on your own program, not only a model benchmark.

What remains unmeasured for your system

MultiLangBench deliberately uses the same database interface across languages so implementations can be compared. That makes the evaluation cleaner; it does not recreate a mainframe estate. OpenAI's migration task checks observable outputs on its hidden cases, while a production service may depend on batch schedules, external files, security controls and interfaces that those cases do not describe. Microsoft and IBM both discuss this broader system context in their linked guidance.

The right executive takeaway is therefore neither “AI can now rewrite all COBOL” nor “benchmarks are irrelevant.” OpenAI has published specific, substantially better model results. Use them as a reason to run a narrowly scoped parity trial with your own rules and dependencies. The trial, not the headline percentage, should decide what moves next.

Sources

• OpenAI, “Advancing coding in low-resource programming languages,” October 19, 2026. Evaluation definitions, language scope, reported model results and limitations. Checked October 21, 2026.
• OpenAI MultiLangBench repository and README. Public benchmark setup and test format. Checked October 21, 2026.
• IBM, “What is COBOL modernization?”, updated April 6, 2026. System scope beyond code translation. Checked October 21, 2026.
• Microsoft, “How We Use AI Agents for COBOL Migration and Mainframe Modernization,” July 9, 2025. Practitioner context for dependencies and redesign. Checked October 21, 2026.