How to Verify AI-Translated COBOL Before It Touches Production

The short answer: don’t trust AI-translated COBOL because it compiles, looks clean or passed code review. Trust it when you can show, with evidence, that it produces the same results as the program it replaces. That takes four things: traceability from each business rule in the COBOL to a requirement, a test and the new code; equivalence tests that run both versions on the same inputs and compare every output field; parallel runs on real production inputs across full business cycles; and data-level reconciliation of what each system actually wrote. Cut over only when the differences are zero or explained and signed off.

Why is this suddenly urgent?

Because translation got fast and cheap this year, and verification didn’t.

In just one week this October:

• On October 12, AWS published a workflow pairing AWS Transform with Devin, Cognition’s coding agent. AWS Transform extracts business rules from COBOL, PL/I and JCL with source-line traceability, and Devin generates the new services, tests and migration scripts from approved specifications.
• On October 13, Rocket Software announced it had acquired Geniez AI to accelerate agentic AI for mainframe operations and modernization.
• On October 16, Semantic Designs’ Modernize Software division announced an approach that uses generative AI for analysis and testing but a deterministic engine for the final translation, arguing that language-model translation is probabilistic and that “errors may compile cleanly and appear correct.”

We agree with that last point, and it’s the reason for this post. We wrote about what AI can and can’t do in COBOL modernization earlier; this one is about the step that decides whether the project is safe: proving the new code is right.

Why isn’t “it compiles and the tests pass” enough?

Because the tests usually came from the same place as the code. If an AI misreads a rule and then writes tests from its own understanding, the tests confirm the mistake.

COBOL also hides behavior in places a line-by-line reading misses:

• Decimal arithmetic. Packed decimal fields (COMP-3), implied decimals and ROUNDED clauses. A translation that uses floating point, or rounds at a different step, can be off by a cent on some records and right on most.
• Truncation and overflow. Moving a value into a smaller PIC field silently drops digits. Modern languages usually throw an error or keep the value.
• Sign and character handling. Signed fields, LOW-VALUES, HIGH-VALUES and EBCDIC sort order behave differently from ASCII and Unicode.
• Dates. Two-digit years, windowing logic and Julian dates hand-rolled decades ago.
• What’s outside the program. JCL step conditions, file sort orders, restart logic and what happens when an input file is empty.

Most of these bugs only show up on a small share of records. That’s exactly why they reach production.

Step 1: How do we trace every business rule?

Before comparing outputs, know what you’re comparing. Build a register of business rules where each rule has an ID, a plain-language statement, the COBOL source lines it came from, the requirement or spec it maps to, the tests that check it, and the new code that implements it.

The AWS Transform workflow above is built this way: each requirement gets an identifier and acceptance criteria, a traceability file records the source lines behind it, and a check confirms every requirement lands in a specification before review. Whatever tools you use, the principle holds. Two rules matter most:

• A business owner confirms the rule, not just the engineer. Extracted rules describe what the mainframe does today. Someone has to say that’s what it should keep doing, or decide to change it on purpose.
• Unexplained rules block, they don’t get guessed. If nobody can say why a rule exists, record the question and hold that piece until someone answers it.

When something fails later, the trace tells you whether the problem is the new code or the rule itself.

Step 2: How do we test equivalence between COBOL and the new code?

Treat the COBOL program as the specification. Feed both versions identical inputs and compare outputs field by field, automatically.

1. Capture real inputs. Pull production input files, transactions and reference tables, masked where needed. Real data has the edge cases nobody remembers.
2. Add generated edge cases. Boundary values for every numeric field, negative and zero amounts, maximum lengths, empty files, invalid dates, duplicate keys. Property-based tests are useful here: rules like “net pay is never negative” should hold for any input.
3. Compare with a typed diff. Compare decoded values, not raw text, and decide up front what counts as equal. A blank field and a null might be the same thing; a one-cent difference is not.
4. Run it in CI. Every regenerated or edited module runs the full comparison before merge. An AI agent that rewrites a function should not be able to merge a change that alters an output.

Aim for coverage of the rules in your register, not just lines of code. A test suite with high line coverage can still skip the year-end branch.

Have AI-translated code and no way to prove it’s right? We build the verification harness: rule traceability, equivalence tests against the COBOL, parallel runs and reconciliation, so you can move jobs off the mainframe with evidence instead of hope. Book a call and we’ll start with one job.

Step 3: How long should old and new run in parallel?

Long enough to cover every business cycle the program handles. For a daily batch job, that usually means at least one month-end. For anything with quarter-end, year-end, tax-table or open-enrollment logic, plan to cover that event or replay a past one.

Run the new system in shadow mode: same inputs, same schedule, outputs written somewhere that doesn’t affect anyone. Compare every run, automatically, and track the mismatch count over time. The goal is a streak of clean runs, not a single good day.

Agree on exit criteria before you start, in writing: how many consecutive clean cycles, which differences are acceptable (for example, a fixed known bug in the old system that the business chose to correct), and who signs off.

Step 4: How do we reconcile the data, not just the reports?

Reports can match while the data underneath doesn’t. Reconcile at three levels:

• Totals. Record counts, control totals and hash totals per file and per run. Fast, and catches big problems.
• Records. Match by key and compare every field. This is where the one-cent rounding differences and truncated fields show up.
• Downstream effects. Check what the next system received: the general ledger postings, the payment file, the extract to the warehouse. A difference in a file nobody reads today can matter at audit time.

Every difference gets classified: a bug in the new code, a bug in the old code the business wants fixed, or an accepted difference. Nothing stays “unexplained.”

Step 5: How do we cut over safely?

One job or one function at a time, never the whole application in one weekend.

• Start with jobs that are well covered by the register and tests and not at the peak of the business calendar.
• Keep the COBOL version able to run for a defined period after cutover, with the data needed to switch back.
• Keep reconciliation running in production for the first cycles after cutover.
• Retire the old job only after the agreed number of clean cycles in production.

This is slower than a big-bang switch. It’s also how you avoid finding the year-end bug in January.

Should we use deterministic translation or AI?

It’s less of a choice than vendors make it sound. Deterministic tools give repeatable output that can be traced to rules. AI is fast and good at reading code and drafting tests. Either way, the output still needs the verification above, because the question isn’t how the code was produced, it’s whether it behaves the same. The three questions in the Semantic Designs announcement are useful for any provider: does running the translation twice produce the same code, can every change be traced to a defined rule, and if the output is wrong, how will the error be found before production?

Frequently asked questions

Can AI-translated COBOL go straight to production?

No. It needs independent verification first: rule traceability, equivalence tests against the original, parallel runs and data reconciliation, with business sign-off on any difference.

What is equivalence testing for COBOL migration?

Running the original COBOL and the translated code on identical inputs and comparing every output automatically, field by field, using real production data plus generated edge cases.

How long should a parallel run last?

At least through every business cycle the program handles, such as month-end, quarter-end or year-end. Agree on the number of clean cycles and the sign-off owner before starting.

What usually goes wrong when COBOL is translated?

Decimal rounding and precision, truncation into smaller fields, sign handling, EBCDIC sort order, date logic and behavior that lives in the JCL rather than the program.

Who should sign off on a translated program?

A business owner for the rules and accepted differences, and engineering for the test and reconciliation evidence. Neither alone is enough.

How we help

We help teams move COBOL workloads off the mainframe in waves, and verification is built into every one. If you’re planning a mainframe migration, or you already have AI-generated code and want to know whether it’s safe, we’re glad to take a look. Grab a time when it suits you.