LLMOps and Vector Databases
When an AI system gives a bad answer, the model is rarely the reason. It is usually that the right passage was never retrieved, and nobody was measuring for it.
- Typical duration
- Three to eight months
- Team shape
- Two to four senior engineers
- Starts with
- A retrieval evaluation
Retrieval is the part worth engineering
Teams that are unhappy with an AI system's answers usually reach for a different model first. It is the wrong lever most of the time: if the passage containing the answer was not in what the model was given, no model can produce it, and a better one will simply produce a more fluent wrong answer.
This engagement is about the layer underneath — how documents are chunked, embedded, indexed and ranked, and how you know any of it is working. That last part is the one most often missing. Without an evaluation harness, retrieval quality is a matter of opinion and a change is a gamble, which is why we build the harness before we tune anything.
What usually goes wrong, and what we do instead
Retrieval problems are invisible without measurement, which is why they persist.
Where retrieval quietly fails
- A fixed chunk size applied to every document type regardless of shape
- Semantic search alone, so exact identifiers and codes are never matched
- No evaluation, so a tuning change is judged by trying a few questions
- Permissions applied after retrieval, so the model saw it anyway
- An index rebuilt manually, and quietly stale between rebuilds
How we build it instead
- Chunking chosen per document type, tested against real questions
- Hybrid search, so identifiers and phrasing are both matched
- An evaluation harness built before any tuning, and run on every change
- Permissions carried into the index and applied at retrieval
- Scheduled incremental indexing, with freshness monitored
What that involves
The layer that decides whether an AI system's answers are good, regardless of which model sits on top.
The evaluation harness first
Before any tuning, something that can tell you it worked.
A scored question set over your own material, run on every change to chunking, embeddings or ranking. Without it, retrieval tuning is guesswork with a confident tone.
Chunking strategy
Chosen per document type and tested, because one size applied to everything is the commonest cause of misses.
Hybrid retrieval
Semantic and keyword search together, so a part number and a paraphrase are both found.
Vector storage
pgvector or a dedicated store, chosen on your scale and on who has to operate it.
Permission-aware retrieval
Access carried into the index, so a model never sees a passage the reader could not.
Indexing pipelines
Incremental and scheduled, with freshness monitored rather than assumed.
Cost and latency budgets
Token spend and response time measured per query type and treated as budgets.
What you end up with
Stated from your side rather than ours.
Retrieval changes you can prove
The harness scores every change, so tuning becomes engineering rather than a sequence of hopeful adjustments.
Permissions that hold at the model
Access is applied at retrieval, so a passage a reader is not entitled to never reaches the prompt in the first place.
Freedom to change model
With retrieval and evaluation as your own infrastructure, swapping provider is a measured decision rather than a rebuild.
Costs you can see and control
Token spend and latency measured per query type, so the expensive path is identifiable rather than an aggregate bill.
How the pieces fit together
Six components, and the model is deliberately the most replaceable of them.
What each part does
- Ingestion: sources pulled on a schedule, with permissions and provenance captured alongside the text.
- Chunking and embedding: split per document type and embedded, with the strategy recorded so a change can be evaluated.
- The index: vectors and keywords together, so exact identifiers and paraphrases are both matched.
- Retrieval and reranking: candidates fetched broadly then reordered, which is usually the cheapest quality gain available.
- The permission filter: applied at retrieval, so the model is never given a passage the reader could not open.
- Evaluation harness: a scored question set run on every change and again in production, so regressions surface.
How the work runs
The same five stages as every engagement, applied to the layer underneath a model.
Diagnose
What is being retrieved today, what is being missed, and whether the complaints are retrieval or model problems.
Decide
The evaluation set, the chunking strategy and the storage choice, written down with what each one costs to operate.
Prove
The harness built and a baseline scored, so every change afterwards has something to be measured against.
Deliver
Pipeline and retrieval improved in slices, each scored against the baseline before the next is scoped.
Hand over
Your team runs the harness and tunes retrieval with us watching, then without us.
What we build it with
Chosen for what has to be operated afterwards, and by whom.
Vector and search
- pgvector
- PostgreSQL
- Elasticsearch
- MongoDB
Pipelines
- Python
- LangChain
- Temporal
- dbt
Models
- Anthropic
- OpenAI
- Gemini
- PyTorch
Platform
- AWS
- Azure
- Docker
- Datadog
What you get, and when
Handed over as it is produced, not assembled at the end.
- A scored question set over your own material, which you keep
- A baseline score for the retrieval you have today
- Which complaints are retrieval problems and which are not
- Source in repositories you own
- Incremental indexing you can re-run over a full corpus
- Permission-aware retrieval, tested against a denied case
- The harness wired into CI, so a change is scored automatically
- Freshness, latency and token-spend dashboards
- The chunking and ranking decisions recorded with their scores
The rest of our AI work
Most AI engagements need one of these underneath them, and some need two.
Related work
Engagements where the quality problem turned out to be underneath the model.
Questions about retrieval and LLMOps
The ones we are asked most often before a first conversation.
Which vector database should we use?
Our AI system gives bad answers. Is this the fix?
What chunk size should we use?
How do you build the evaluation set?
How do we stop it returning things people should not see?
Does this tie us to one model provider?
AI system giving answers you cannot trust?
Tell us what it is getting wrong. We will find out whether the problem is the model or what it was given.