Loading
Preparing infrastructure view...
Home / Infrastructure design / Data and retrieval
The layer that sits behind the models
Ingestion, embedding pipelines, a vector store and retrieval, built so the answers a model gives are grounded in your data rather than its memory.
What it is
Data and retrieval, in plain terms
Ingestion, embedding pipelines, a vector store and retrieval, built so the answers a model gives are grounded in your data rather than its memory.
Ingestion pipelines
Documents and records pulled in, cleaned and chunked.
Vector and feature stores
Storage sized for the corpus and the query pattern.
Retrieval
Hybrid search and reranking, tuned against real questions.
Permissions carried through
A user only retrieves what they were already allowed to read.
How it works
How data and retrieval is delivered
- 01Chunk with meaningBoundaries follow document structure. Fixed-size chunking is fast to build and expensive to live with.
- 02Carry permissions throughAccess tags travel with the chunk, so retrieval filters by what the user may already read.
- 03Embed incrementallyFull re-embedding is scheduled and cheap, so changing model does not become a project.
- 04Hybrid then rerankKeyword and vector search together, then a reranker, because either alone returns confident nonsense.
- 05Evaluate on real questionsA held-out set of questions your users actually ask, scored before and after every change.
In the field
Data and retrieval on site
What the work looks like once it is on the floor.
Detail
What is included
| Sources | Files, databases, wikis, ticket systems and object storage. |
|---|---|
| Embedding | Batch and incremental, with re-embedding when the model changes. |
| Store | Vector store sized to corpus and latency target. |
| Retrieval | Hybrid keyword and vector search with reranking. |
| Evaluation | A test set of real questions, scored before and after changes. |
Watch
Watch: how we approach data and retrieval platforms
Watch
FILMFilm Data and retrieval platforms walkthrough · 2:05
Support
Common questions
How is it priced?↓
Fixed price against a defined scope. Ongoing operation is quoted separately as a monthly service.
Do you supply the hardware?↓
We specify it and can procure it, but it stays on your balance sheet.
Can a partner sell this under their own name?↓
Yes. Design and build are delivered white-label where a partner holds the customer.
Next step
Tell us what you are building and we will tell you what it needs.
An hour with our engineers, no charge. Bring a floor plan, a workload, or just the problem.