Loading
Preparing infrastructure view...
Home / Infrastructure design / AI training
Multi-node GPU that stays busy
Training is where a badly designed cluster shows up fastest. Checkpointing, scheduling and topology-aware placement, so the accelerators are not waiting on the network.
What it is
AI training, in plain terms
Training is where a badly designed cluster shows up fastest. Checkpointing, scheduling and topology-aware placement, so the accelerators are not waiting on the network.
Topology-aware placement
Jobs mapped to the fabric rather than scattered across it.
Checkpointing
Long runs survive a node failure instead of starting again.
Scheduling and queues
Fair share across teams, with accounting per project.
Power and cooling sized for it
The rack design follows the cluster, not the other way round.
How it works
How ai training is delivered
- 01Feed rate before flopsWe size data loading and storage throughput first. A starved GPU is the most expensive idle asset you own.
- 02Placement on the wireJobs are mapped to the physical topology so all-reduce traffic stays inside a rail instead of crossing the spine.
- 03Checkpoint like you mean itCheckpoint frequency and write bandwidth are designed together, so a node failure costs minutes.
- 04Gang schedulingA distributed job gets all its ranks or none. Partial allocations quietly waste whole nodes.
- 05Measure the runCost per run and energy per run are reported from the first job, not added once somebody asks.
In the field
AI training on site
What the work looks like once it is on the floor.
Detail
What is included
| Scale | From four nodes to multi-rack, with the fabric sized to the model. |
|---|---|
| Interconnect | InfiniBand or RoCE, chosen against the communication pattern. |
| Storage | Throughput sized so checkpoint writes do not stall the run. |
| Scheduling | Slurm or Kubernetes with gang scheduling. |
| Measurement | Utilisation, cost per run and energy per run, reported from day one. |
Watch
Watch: how we approach ai training clusters
Watch
FILMFilm AI training clusters walkthrough · 2:05
Support
Common questions
How is it priced?↓
Fixed price against a defined scope. Ongoing operation is quoted separately as a monthly service.
Do you supply the hardware?↓
We specify it and can procure it, but it stays on your balance sheet.
Can a partner sell this under their own name?↓
Yes. Design and build are delivered white-label where a partner holds the customer.
Next step
Tell us what you are building and we will tell you what it needs.
An hour with our engineers, no charge. Bring a floor plan, a workload, or just the problem.