Benchmarks & economics
The case for Collimate is quantitative: spawn an isolated VM in well under a millisecond, hold a thousand of them on one host for kilobytes each, and watch per-VM cost drop as you scale. Here are the measured numbers and what they mean for the cost of a rollout.
Spawn latency
End-to-end time to spawn a fresh, fully-restored microVM, measured over 1,000 iterations on a GCP n2-standard-8 (8 vCPU, 32 GiB):
| Percentile | Latency |
|---|---|
| Min | 0.987 ms |
| P50 | 1.175 ms |
| P95 | 1.347 ms |
| P99 | 1.454 ms |
| Max | 1.736 ms |
On AWS c8i.xlarge (the production-target platform) the same spawn measures ~0.65 ms P50 / ~0.996 ms P99.
Concurrent scaling
All VMs created simultaneously. Per-VM latency and memory both improve with scale, because the more VMs you run, the more memory stays shared:
| Concurrent VMs | Wall-clock | Per VM | Memory / VM |
|---|---|---|---|
| 10 | 8.6 ms | 856 µs | 506 KB |
| 100 | 84 ms | 841 µs | 174 KB |
| 1,000 | 1,225 ms | 1,225 µs | 141 KB |
A single 8-vCPU host sustained 1,000+ concurrent VMs; on memory alone, the per-VM footprint would fit roughly 100K VMs in a large box's RAM, though CPU scheduling and kernel limits bind first; we've measured 1,000+.
Versus the runtimes teams actually use
Same workload, mainstream sandbox runtimes. Latency is spawn P50; memory is steady-state per sandbox.
| Runtime | Spawn P50 | Memory / sandbox | vs Collimate |
|---|---|---|---|
| Collimate | ~0.65-1.2 ms | ~141 KB | baseline |
| Daytona | ~27 ms | ~50 MB | 23× slower · 290× heavier |
| E2B | ~150 ms | ~128 MB | 128× slower · 740× heavier |
| microsandbox | ~200 ms | ~50 MB | 170× slower · 290× heavier |
| Docker (runc) | ~3,350 ms | ~4 MB | shared-kernel container |
| gVisor (runsc) | ~2,886 ms | n/a | shared-kernel sandbox |
Sandbox-runtime figures (E2B, Daytona, microsandbox) are vendor-published; container figures (runc, gVisor) were measured on the primary host and are a different isolation class, listed for latency context.
Time-to-interactive
VM spawn plus a real round-trip of work:
| Workload | Total | of which spawn |
|---|---|---|
| VM spawn + echo | 5.8 ms | 1.1 ms |
| VM spawn + echo (via API) | 8.4 ms | 1.3 ms |
| VM spawn + cold Python (sum 1M) | 353 ms | 1.3 ms |
The spawn is a rounding error. What you wait for is your own workload, which is exactly how it should be.
What a rollout costs
Density is the whole economic argument. Memory is usually what caps how many environments you can run at once:
1,000 concurrent sandboxes, memory footprint: Collimate ▏ ~140 MB (141 KB × 1,000) Daytona ████████████████████████████████ ~50 GB E2B ██████████████████████████████████████████████████████████████████ ~128 GB
Because VMs share their template copy-on-write, you pay for divergence, not for environment count. That collapses the cost of the rollout phase: GRPO-style group sampling, best-of-N, and wide tree search stop being memory-bound, so the same budget buys far more exploration.
Methodology
Honest about what's measured. Latency, scaling, and memory above are measured results. Numbers on other pages tied to tree search (e.g. replay-vs-fork overhead and per-branch memory for a browser workload) are analytical or projected; the end-to-end success-rate benchmark is still in progress, and we mark projections as such.