Generative AI Solutions
The demo is easy and the demo is not the product. What decides whether this works is retrieval quality, evaluation, and knowing which answers a person still has to sign.
- Typical duration
- Three to nine months
- Team shape
- Two to four senior engineers
- Starts with
- An evaluation, not a build
We start by finding out whether it works on your data
Generative systems are unusually easy to prototype and unusually hard to assess. A convincing demo tells you almost nothing about how the system behaves on the awkward eighty per cent of real inputs, and that gap is where most of these projects quietly fail.
So the first deliverable is an evaluation rather than a model: a scored test set built from your own material, against a definition of accurate that you agreed in advance, and a written recommendation that is allowed to say no. If it says yes, the build that follows is mostly retrieval engineering and human-in-the-loop design — because that is where answer quality actually lives, not in the choice of model.
What usually goes wrong, and what we do instead
The pattern is consistent enough that we now refuse to start with a build.
Where generative projects fail
- A demo on clean examples, then real inputs that look nothing like them
- No agreed definition of a correct answer, so quality is a matter of opinion
- Answers with no citation, so nobody can check whether they are right
- A model given authority to act where it should only have drafted
- Quality regressions after launch that nobody is measuring for
How we build it instead
- A scored evaluation on your own data before anything is built
- Accuracy defined and agreed in writing, per task, in advance
- Retrieval with citations, so every answer traces to a source
- A written line between what the system drafts and what it decides
- Evaluation running in production, so a regression is visible early
What that involves
The work that decides whether a generative system changes an outcome or only impresses in a meeting.
Evaluation before build
The first deliverable is a decision, not a model.
We measure whether a model can do the job on your own data, against a definition of accurate that you agreed in advance. The result is a written recommendation, and it is allowed to say no.
Retrieval
Chunking, indexing and ranking over your own material, which is where most answer quality is won.
Assistants and drafting
Interfaces that produce a first draft a person edits, rather than an answer they have to trust.
Guardrails
Refusals, scope limits and citation requirements, tested rather than described in a prompt.
Human-in-the-loop design
Where the system drafts, where it acts, and where a person still has to sign.
Integration
Placed inside the tool people already use, rather than beside it as a second place to look.
Evaluation in production
Accuracy tracked after launch, so a quiet regression is found before a customer finds it.
What you end up with
Stated from your side rather than ours.
An answer before the investment
The evaluation tells you whether this can work on your data, and it costs a fraction of finding out by building.
Answers somebody can check
Every response cites what it drew on, so a wrong answer is traceable rather than mysterious.
A defensible authority line
What the system decides and what a person signs is written down, which is what a regulator or an auditor will ask about.
Quality you keep measuring
Evaluation runs in production, so a model or data change that degrades answers shows up on a dashboard.
How the pieces fit together
The model is one component among six, and it is rarely the one that decides whether the answers are good.
What each part does
- Ingestion: your material, chunked and indexed on a schedule, with the source and its permissions carried through.
- Retrieval: ranking that finds the right passages for a question — the component that most determines answer quality.
- The model: swappable behind an interface, because the right provider changes and you should not be rebuilt when it does.
- Guardrails: scope limits, refusal behaviour and the requirement to cite, enforced in code rather than asked for in a prompt.
- The human boundary: which answers are drafts for a person to edit and which the system is permitted to act on.
- Evaluation: a scored test set run before release and again in production, so a regression is visible rather than reported.
How the work runs
The same five stages as every engagement, applied to a system whose output is probabilistic.
Diagnose
What decision this is meant to change, what accurate means for it, and what material exists to retrieve from.
Decide
The evaluation is run and the recommendation written. This is where a no costs you weeks rather than a year.
Prove
One task in production with real users, scored against the test set and with its human boundary in place.
Deliver
Tasks added in slices, each evaluated before the next is scoped.
Hand over
Your team runs the evaluation and the pipeline with us watching, then without us.
What we build it with
Chosen for what has to be explained afterwards, and for who has to run it once we have gone.
Models
- Anthropic
- OpenAI
- Gemini
- PyTorch
Retrieval
- LangChain
- pgvector
- Elasticsearch
- MongoDB
Services
- Python
- TypeScript
- Node.js
- PostgreSQL
Platform
- AWS
- Azure
- Docker
- Datadog
What you get, and when
Handed over as it is produced, not assembled at the end.
- A scored test set built from your own material, which you keep
- The agreed definition of accurate, per task
- A written recommendation, which is allowed to be 'not yet'
- Source in repositories you own
- An ingestion and retrieval pipeline you can re-run
- The model behind an interface, so a provider can be changed
- The human boundary, written down per task
- Guardrail tests, including the refusals
- Production evaluation dashboards with alerting on regression
The rest of our AI work
Most AI engagements need one of these underneath them, and some need two.
Related work
Engagements where the honest answer mattered more than the impressive one.
Questions about generative AI
The ones we are asked most often before a first conversation.
How accurate will it be?
What stops it making things up?
Does our data get used to train someone's model?
Which model should we use?
What does it cost to run?
What if the evaluation says it will not work?
Wondering whether a model can do this?
Tell us the task and what accurate would mean. We will measure it on your data before anyone builds anything.