Loading
Preparing infrastructure view...
Home / Infrastructure design / AI inference
Serving sized to a latency target
Inference is a different problem from training. Batching, caching and autoscaling designed against the latency you actually promised your users.
What it is
AI inference, in plain terms
Inference is a different problem from training. Batching, caching and autoscaling designed against the latency you actually promised your users.
Model serving
Endpoints with versioning, canary releases and rollback.
Batching and autoscaling
Throughput without breaking the latency budget.
Caching
KV and result caching where it earns its keep.
Cost per request
What one answer costs, measured rather than estimated.
How it works
How ai inference is delivered
- 01Start from the budgetp50 and p95 targets are agreed first. Every later decision is measured against them.
- 02Batch inside the budgetContinuous batching raises throughput, but only up to the point where it breaks the latency you promised.
- 03Cache what repeatsKV and result caching are applied where the traffic shape justifies them, not everywhere by default.
- 04Scale on the right signalAutoscaling reacts to queue depth and latency. CPU utilisation is a lagging and misleading trigger.
- 05Ship models safelyCanary and shadow traffic, a registry, and a rollback that has been used at least once before it is needed.
In the field
AI inference on site
What the work looks like once it is on the floor.
Detail
What is included
| Serving | Open models or your own, on GPU or CPU as the workload warrants. |
|---|---|
| Scaling | Autoscaling against queue depth and latency, not just CPU. |
| Routing | Canary and shadow traffic for model changes. |
| Observability | Latency percentiles, error rates and token throughput. |
| Governance | Prompt and output logging where the regulator expects it. |
Watch
Watch: how we approach ai inference platforms
Watch
FILMFilm AI inference platforms walkthrough · 2:05
Support
Common questions
How is it priced?↓
Fixed price against a defined scope. Ongoing operation is quoted separately as a monthly service.
Do you supply the hardware?↓
We specify it and can procure it, but it stays on your balance sheet.
Can a partner sell this under their own name?↓
Yes. Design and build are delivered white-label where a partner holds the customer.
Next step
Tell us what you are building and we will tell you what it needs.
An hour with our engineers, no charge. Bring a floor plan, a workload, or just the problem.