AI Development

Legacy System AI Modernization

The rules in that system encode twenty years of hard-won knowledge. The right move is to keep them running while a model learns to beat them, not to switch them off and hope.

Typical duration: Six to fifteen months
Team shape: Three to five senior engineers
Starts with: Shadow running, not replacing

A model's decisions compared against an existing rules engine on the same cases

The old rules are the baseline, and they are usually good

A decision engine that has been tuned by domain experts for a decade is a formidable thing to beat. It is also brittle, undocumented and impossible to change — which is why it is being replaced, and why replacing it wholesale so often goes badly.

We run the model in shadow instead. Every case goes through both the existing rules and the new model, the two are compared on real outcomes, and nothing changes for anyone until the evidence says it should. Where the model wins, that decision moves. Where the rules win — and on some decisions they simply do — the rules stay, and the honest finding is recorded rather than argued around.

What usually goes wrong, and what we do instead

The failure is not the model. It is switching off a baseline nobody measured first.

Where these projects fail

The rules replaced before anyone measured how well they performed
Rules treated as documentation, when the code and the docs disagree
A model judged on aggregate accuracy, hiding the cases that matter most
Experts excluded, so the knowledge in the rules is lost with them
No way back once the old engine has been switched off

How we run it instead

The existing rules characterised and scored before anything is built
Behaviour taken from the running system, not from its documentation
Performance compared per case type, not as a single accuracy figure
Domain experts in the loop, adjudicating where the two disagree
The old engine kept live and authoritative until a decision moves

What changes, in practice

Not a technology list. The things that are different for the people who own and operate the decision.

The system today

Accurate, and frozen.

Rules accumulated over years, with no record of why most exist
A change requires the one person who understands the engine
Nobody has measured how well it performs on current cases
Edge cases are handled by a manual override nobody logs
New patterns in the data need a rule somebody has to write
There is no way to test a change except to release it

The system after

Same decisions, measurable and changeable.

Rules characterised in tests, so what they do is readable
Any engineer can change a decision path safely
Performance scored continuously against real outcomes
Overrides captured as signal rather than lost as workarounds
New patterns are learned rather than hand-encoded
A change is scored in shadow before it affects anyone

What that involves

The work that decides whether judgement can be added to an old system without losing what it already knows.

Shadow running

Both engines, every case, nothing switched off.

The model runs beside the existing rules on real cases and the two are compared on real outcomes. A decision moves only when the evidence says it should, and some decisions never do.

Rule characterisation

Tests written from the running engine, because the code and the documentation rarely agree.

Per-case-type scoring

Performance compared by case type, since an aggregate figure hides exactly the cases that matter.

Expert adjudication

Where the two engines disagree, the people who own the decision say which was right.

Slice-by-slice cutover

One decision path moves at a time, reversibly, with the old engine still live behind it.

Integration preservation

Everything downstream keeps working, including the consumers nobody currently owns.

Scoring in production

Decisions compared with outcomes as they arrive, so drift is visible before a quarter is.

How the pieces fit together

Six components, and the point of the arrangement is that the fifth one can be switched back at any moment.

Cases flowing through both an existing rules engine and a model, compared by an adjudication layer, with a router deciding which is authoritative per decision — Stand-in artwork. The numbered legend describes the system rather than this picture.

What each part does

The case: what arrives, captured with everything known at the moment the decision has to be made.
The existing rules: still running, still authoritative, and now characterised by tests so its behaviour is readable.
The model: scored on the same cases in shadow, producing a decision and the reasoning behind it.
Adjudication: disagreements surfaced to the domain experts, whose verdicts become the scored evidence.
The router: which engine is authoritative, per decision path, changeable in both directions without a release.
Outcome feedback: what actually happened, matched back to both engines — the loop that decides what moves and what stays.

How the programme runs

The same five stages as every engagement, applied to a system whose decisions matter.

Diagnose

What the rules actually do, how well they perform on current cases, and which decision path is worth attempting first.

Decide

The comparison method, the case types and what would count as beating the baseline, agreed in writing.

Prove

The model run in shadow on real cases, scored, and adjudicated with your experts before anything is switched.

Deliver

Decision paths moved one at a time, each reversible, with the rules still live behind them.

Hand over

Your team scores, retrains and moves a decision path with us watching, then without us.

What we work with

The starting point is whatever the old system runs on. This is what we add beside it.

Modelling

Python
PyTorch
scikit-learn
Anthropic

Seams

REST
Kafka
Temporal
Java

Data

PostgreSQL
Snowflake
dbt
Elasticsearch

Platform

AWS
Azure
Docker
Datadog

The rest of our AI work

Most AI engagements need one of these underneath them, and some need two.

All AI development services

Generative AI Solutions

Assistants and drafting tools with the retrieval and guardrails that make their answers accountable.

Explore Generative AI Solutions

Predictive Analytics

Forecasts scored against what actually happened, and wired into the decision they exist to change.

Explore Predictive Analytics

Document Intelligence

Extraction from the documents you really receive, including the scanned and the badly filled in.

Explore Document Intelligence

AI Process Automation

The steps a model can take on its own, and the ones a person still has to sign off.

Explore AI Process Automation

LLMOps and Vector Databases

Indexing, retrieval and evaluation pipelines, which is where most answer quality actually lives.

Explore LLMOps and Vector Databases

AI Integration and Deployment

Putting a model behind an existing system, with the fallbacks for when it is unavailable.

Explore AI Integration and Deployment

Related work

Engagements where an old system knew more than anyone could explain.

All case studies

Brindley WaterUtilities

Retiring a mainframe without a big-bang cutover

hours of downtime

Read the case study

Thackray HealthHealthcare

Making patient data available without moving it

faster record retrieval

Read the case study

What we work towards

ILLUSTRATIVE, pending sign-off. These are the measures this work is judged on and the direction they should move — not results from a named engagement. They will carry real figures or be removed before launch.

Baseline performance

Never measuredScored continuously

Time to change a decision rule

One person, weeksAny engineer, days

Testing a change

Release it and watchScored in shadow first

Manual overrides

Unlogged workaroundsCaptured as training signal

Rollback for a moved decision

A releaseA routing change

Questions about this work

The ones we are asked most often before a first conversation.

What if the model does not beat our rules?

Then the rules stay, and you have learned something valuable for a fraction of what finding out the hard way costs. It happens more often than the industry admits: a decision engine tuned by experts over a decade is a strong baseline, particularly on the common cases. Usually the finding is partial — the model wins on some case types and loses on others — and only the paths where it wins should move.

Nobody here fully understands the existing rules.

That is the normal starting position, and it is why characterisation comes before anything else. We derive the behaviour from the running system rather than from documentation, because on a system of this age the two reliably disagree. That characterisation is often valuable on its own, even if the AI work is subsequently deferred.

Our decisions have to be explainable. Does that rule this out?

No, but it constrains the modelling, and rightly so. Where a decision is regulated or materially affects a person, we favour inherently interpretable models even at some cost in accuracy, because a decision you cannot defend is one you cannot deploy. The comparison against the existing rules also helps here: an explanation in terms of the factors the old engine used is one your experts already recognise.

How long does shadow running go on?

Until you have enough adjudicated cases to be confident, which depends on your volume and on how often the interesting cases occur — for a high-volume decision that can be weeks, for a rare one it may be many months. We agree the evidence threshold before starting rather than deciding it when the results are in, because deciding afterwards is how people talk themselves into a switch.

How much of our experts' time does this need?

Real amounts, mostly in adjudication, and it is the input that most determines the outcome. Where the two engines disagree, only your people can say which was right, and those verdicts are the evidence everything else rests on. We keep it to batches of the genuinely ambiguous cases rather than everything, but a project where the experts cannot participate is one we would advise against starting.

Do the old rules ever get switched off?

Per decision path, and only once that path has been authoritative on the model for long enough to trust — and some paths never move, which is a legitimate end state rather than an incomplete project. Decommissioning what remains is its own piece of work at the end, done properly, because an old engine left running because nobody was sure is a cost and a risk for years.

Old system that knows more than anyone can explain?

Tell us what decision it makes and how well it makes it. We will find out whether a model can do better.

Start a project