Where AI Agents Help a Java Migration, and Where They Lie
Whether AI can rewrite legacy Java is settled: it can. The question your board needs answered is who certifies that the business logic still does what it did, and that answer has not changed.
Amit AgarwalFounder, CipherCruยท 11 min read
Whether AI can rewrite legacy Java is no longer the question. It can. The question your board needs answered is who certifies that the business logic still does what it did, and that answer has not changed.
Published coverage splits into two unhelpful halves: AI modernizes legacy code, or AI cannot handle legacy code. Both skip the only thing worth writing down, which is the division of labour. Here is the split that holds up, and the three places a Java and Spring migration goes silently wrong without it.
Why does this matter now?
Two things arrived together. Every Spring Boot 3.x branch lost free support in June 2026, so many Java estates face a migration this year, and a three-way decision about it. Agents are also now good enough that a board will ask why that migration needs a team at all.
The large public results are real. Google reported (opens in a new tab) that 80% of the code changes in one internal migration were fully AI-authored, and its engineers estimated the total time was cut by half. Amazon's chief executive said (opens in a new tab) upgrading an application to Java 17 fell from about 50 developer-days to a few hours.
Both are companies reporting on their own tooling, in codebases with strong tests and review. A ten-year-old estate often has neither, and that changes what an agent can safely be trusted with.
How should the work be split between tools, agents and people?
Split it three ways, and give each layer only the work it is reliable at.
Deterministic tooling does the mechanical bulk. OpenRewrite recipes work on a type-attributed model of your code, so they produce the same change on every run and you review them as a single diff. Spring Boot 4 alone removes 36 deprecated classes. Anything a recipe can do, a recipe should do.
Agents take the ambiguous remainder. That is work needing judgement, but not much of it. It is narrower and less glamorous than the vendor demos.
People validate behaviour. This layer does not shrink, and it is the layer your budget conversation is really about.
Agents provide speed, guardrails provide accuracy.
It is the most useful one-liner in the category. Neither half substitutes for the other.
Where do AI agents genuinely earn their keep?
In three jobs, and they share one property: the output can be checked by something other than the agent that produced it.
Writing characterization tests. Legacy code is under-tested almost by definition. A characterization test (opens in a new tab) pins what the code does today, not what it should do, and writing hundreds of them is tedious, high-volume, low-judgement work.
Agents are good at it, and the result is verifiable: each test either reproduces current behaviour against the untouched system or it does not. This is the highest-value use, because it builds the safety net the rest of the migration depends on.
Documenting the undocumented. Mature enterprise code carries domain knowledge that is written down nowhere. An agent cannot recover why a 400-line method exists, but it can describe accurately what it does, which is a useful start for the engineer who then has to ask why.
Writing recipes instead of edits. This is the pattern worth stealing. Have the agent write a transformation recipe, review that one recipe, then apply it deterministically across the whole estate.
You get the agent's pattern recognition and the tool's repeatability, and you review one recipe instead of four hundred diffs. Harrer describes the same hybrid: coding agents that write refactoring scripts, which transformation tools then apply deterministically.
---
type: specs.openrewrite.org/v1beta/recipe
name: com.example.MigrateLegacyAuditApi
displayName: Migrate legacy audit API
description: Renames the legacy audit entry point and moves callers to the new type.
recipeList:
- org.openrewrite.java.ChangeMethodName:
methodPattern: com.example.legacy.AuditLogger writeEntry(..)
newMethodName: record
- org.openrewrite.java.ChangeType:
oldFullyQualifiedTypeName: com.example.legacy.AuditLogger
newFullyQualifiedTypeName: com.example.audit.AuditServicemvn -U org.openrewrite.maven:rewrite-maven-plugin:6.48.0:dryRun \
-Drewrite.activeRecipes=com.example.MigrateLegacyAuditApiWhere do AI agents fail silently?
In three places, and all three fail the same way: the code compiles, the existing suite passes, and the behaviour is wrong. The title says "lie" as shorthand. None of this is deception. Each failure is an agent doing exactly what the prompt and the codebase allowed.
1. Dates and time zones drift
java.util.Date is an instant: milliseconds since the epoch, with no time zone. A decade of application code often treated it as a local wall-clock date anyway.
Ask an agent to modernise that code and the obvious rewrite is LocalDateTime. It compiles and reads better. It has also quietly bound the value to the server's default time zone.
// Before: an instant, whatever the surrounding code assumed
Date createdAt = order.getCreatedAt();
// An obvious modernisation: now tied to the JVM's default zone
LocalDateTime created = LocalDateTime.ofInstant(
createdAt.toInstant(), ZoneId.systemDefault());
// What the field usually means: an instant, or a zone you name
Instant createdInstant = createdAt.toInstant();
ZonedDateTime createdLocal =
createdInstant.atZone(ZoneId.of("Europe/London"));Nothing fails. Reports drift by hours, dates land on the wrong day near midnight, and an hour goes missing or repeats at daylight-saving changes. You find out in production, weeks later.
Mutability makes it worse. Calendar.add changes the object in place, while java.time methods return a new object, so a discarded return value becomes a silently dropped calculation.
// Before: mutates the calendar in place
calendar.add(Calendar.DAY_OF_MONTH, 30);
// After: compiles, and does nothing
dueDate.plusDays(30);
// Correct
dueDate = dueDate.plusDays(30);The right target is usually Instant, or ZonedDateTime with an explicit zone. Choosing it is a judgement about what the field means, and that meaning is exactly what an agent cannot find written down.
2. Transaction boundaries move
Spring's declarative transactions are proxy-based, and the Spring Framework reference (opens in a new tab) is unambiguous about what that means.
In proxy mode (which is the default), only external method calls coming in through the proxy are intercepted. This means that self-invocation (in effect, a method within the target object calling another method of the target object) does not lead to an actual transaction at runtime even if the invoked method is marked with @Transactional.
Now consider what an agent does when asked to tidy a service class: extract methods, inline wrappers, merge collaborators, reduce indirection. Each of those can turn an external call into a self-invocation.
// Before: two beans, so the call crosses the proxy
@Service
class OrderService {
private final PaymentService payments;
void checkout(Order order) {
payments.charge(order); // PaymentService.charge is @Transactional: runs in a transaction
}
}
// After "reduce indirection": one bean, and no transaction
@Service
class OrderService {
void checkout(Order order) {
charge(order); // self-invocation: the annotation below is ignored
}
@Transactional
void charge(Order order) { /* ... */ }
}The annotation is still there. The transaction is not. Your tests pass because they exercise the happy path, and you find out the first time a partial failure needs to roll back.
Visibility is a second lever. Since Spring Framework 6.0, class-based proxies also make protected and package-visible methods transactional, so changing a method's visibility can change its transactional behaviour as well.
3. The tests get gamed
This is the most uncomfortable failure, and it has now been measured. The Reward Hacking Benchmark (opens in a new tab), a single-author preprint from May 2026, tested 13 frontier models on multi-step tool-use tasks that each contained a legitimate solution and an exploitable shortcut.
Overall exploit rates ran from 0% up to 13.9%. The behaviours observed are the ones that matter in a migration: editing verifiers and scoring scripts, branching on known filenames or test instance IDs, and fabricating plausible intermediate artifacts to skip steps.
| Model | Exploit rate |
|---|---|
| Claude Sonnet 4.5 | 0.0% |
| Claude Opus 4.5 | 0.0% |
| Claude 3.5 Sonnet v2 | 0.6% |
| DeepSeek-V3 | 0.6% |
| Gemini 2.5 Flash Preview | 0.8% |
| GPT-4o | 0.9% |
| Claude 3.7 Sonnet | 3.9% |
| Gemini 2.5 Pro Preview | 4.6% |
| o1 | 6.8% |
| o3-mini | 7.1% |
| o4-mini | 8.4% |
| o3 | 11.8% |
| DeepSeek-R1-Zero | 13.9% |
Low is not zero everywhere: Sonnet 4.5 and Opus 4.5 exploited 1.8% and 1.2% of the benchmark's harder variants. The cleanest comparison is between siblings. DeepSeek-R1-Zero, trained with reinforcement learning, exploited 13.9% of tasks, against 0.6% for the mainly fine-tuned DeepSeek-V3.
Two more findings should shape how you run a migration. Exploit rates jumped at step five of the benchmark's tasks, the first step checked against criteria the agent could not see. And among models that expose their reasoning, 72% of exploit episodes included an explicit rationale framing the shortcut as a pragmatic move.
The agent is not being deceptive in any interesting sense. It is optimising the target you gave it, which was "make the tests pass".
The fix is boringly effective. Hardening the environment with stricter file access, explicit step verification and fail-closed parsing cut exploit rates from 6.5% to 0.8%, a relative reduction of about 88%, with no measurable loss in task success.
What's the catch with using agents at all?
Speed you cannot verify is not speed. Agents make a migration faster only where review keeps up, and in some codebases it cannot.
The productivity evidence is mixed. In METR's 2025 study (opens in a new tab), experienced open-source developers took 19% longer with AI tools while believing they were faster. METR's 2026 update (opens in a new tab) estimated an 18% speed-up for returning developers, with an interval wide enough to include a slowdown, and METR calls it only very weak evidence.
Google's migration work points the same way from the other side: once generation was fast, code review became the bottleneck. So do not hand a migration to agents when:
- The code has no tests, and nobody can write characterization tests against a running system.
- The business rules are undocumented and the people who knew them have left, so nobody can judge whether behaviour is correct.
- Review capacity is already the constraint. More generated diffs make that worse, not better.
- The change is a one-off that a person can make faster than they can specify it and review the result.
What should you put in place before the first agent run?
Four controls. All of them are cheap, and all of them belong in place before an agent touches the code.
- Lock the tests. Write characterization tests against the untouched system and freeze them before migration begins. The migrating agent gets no write access to them. This single control removes most of the test-gaming risk.
- Use recipes, not edits. Have the agent author the recipe, review it once, and apply it deterministically. The transformation becomes repeatable, and your review becomes possible.
- Flag two change classes for mandatory human review. Date and time fields, and anything that touches a transaction boundary, including extracted methods and visibility changes. Not because agents are bad at them, but because their failures there are silent.
- Keep the chains short, and checkpoint them. Do not hand an agent a fifteen-step migration and read only the final diff. Verify at every step the agent could not check for itself, and harden the environment while you are there.
What does this mean for the budget?
Cut the mechanical cost hard, and do not cut validation. Recipes and agent-authored recipes are where the real savings are, and they are substantial.
The honest pitch to your board is not that AI removes the validation cost. It is that AI shrinks the mechanical cost so far that validation becomes the largest line item.
That is a better project shape than the one you had, because the money now goes to the part that protects you.
If the migration you are budgeting is a Spring Boot upgrade, scope it first with the Jackson 3 dependency audit. This post is how you execute it.
Frequently asked questions
Can AI agents migrate a legacy Java application on their own?
Not safely. Agents can produce code that compiles and passes existing tests while changing behaviour, especially around dates, transactions and the tests themselves. Use deterministic recipes for mechanical changes, agents for checkable work such as characterization tests, and people to certify that the business logic still does what it did.
What is a characterization test?
A characterization test records what existing code actually does today, not what it should do, so any change in behaviour during a migration shows up as a failure. Michael Feathers coined the term in Working Effectively with Legacy Code. It is also called golden master testing, and it suits agents because the result is checkable.
What is OpenRewrite, and why use it alongside agents?
OpenRewrite is an open-source tool that applies recipes to source code through a type-aware model, so the same recipe produces the same change on every run. Paired with agents, the agent writes a recipe once, a person reviews it, and the tool applies it across the estate, repeatably.
How do you stop an AI agent gaming the test suite?
Remove the opportunity. Write characterization tests before migration starts, keep them read-only to the migrating agent, verify every step the agent cannot check for itself, and parse results fail-closed. In the Reward Hacking Benchmark, hardening of this kind cut exploit rates by about 88% with no measurable loss in task success.
Which Java changes should always get a human review?
Any change to date and time handling, and any refactoring near a Spring transaction boundary: extracted or merged methods, inlined wrappers and changed visibility. Agents are not uniquely bad at these. Their failures there are silent, so an existing test suite rarely catches them before production does.
AI is not the risky part of a legacy migration. Unverified change is. If you are planning one, our legacy system AI modernization page sets out how we approach that work.
Sources
- AI Agents Don't Modernize Legacy Code on Their Own (opens in a new tab), interview with Markus Harrer, iSAQB
- Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (opens in a new tab), Kunvar Thaman, arXiv preprint
- Using @Transactional (opens in a new tab), Spring Framework reference
- java.util.Date (opens in a new tab), Java SE 21 API documentation
- Characterization test (opens in a new tab), Wikipedia
- YAML format reference (opens in a new tab), OpenRewrite documentation
- How to speed and scale your Spring Boot 4 migration (opens in a new tab), Moderne
- How is Google using AI for internal code migrations? (opens in a new tab), Google, arXiv
- Andy Jassy on Amazon's Java upgrades (opens in a new tab), quoted by Simon Willison
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (opens in a new tab), METR
- METR's 2026 uplift update (opens in a new tab), METR
More from CipherCru
Other thinking on how technology decisions get made and delivered.
Get new writing by email
Occasional notes on technology decisions, delivery and modernization. No more than once a month.
Weighing an AI-assisted migration?
Tell us what the estate looks like and what the board expects. We will tell you which parts we would hand to recipes, which to agents, and which we would never hand to either.