Every coding agent I run wants to execute code it just wrote. Containers are the default answer and the wrong one: a shared kernel is a shared attack surface, and the code running inside was written by a model thirty seconds ago. MicroVMs fix that. Firecracker boots a real Linux kernel in a hardware-isolated VM in well under a second, and every sandbox vendor is built on it or on something like it.
There are a dozen tutorials on booting a Firecracker VM. There is almost nothing on what it takes to run a fleet of them behind an API. So I built one. This is what I learned, with numbers.
The code is at github.com/avirajkhare00/sandboxd. One Go binary, two dependencies, about 600 lines.
What it does
The service exposes four endpoints an agent harness calls: create a sandbox, run a command in it, upload a file, destroy it. Under the hood each sandbox is:
- a Firecracker microVM restored from a golden snapshot, so it comes up with Python, Node, and git already loaded
- running inside its own network namespace with a tap device, so every VM sees the same interface name and IP
- rooted on a copy-on-write clone of a base ext4 image
- reachable over vsock, where a 100-line agent inside the guest accepts newline-delimited JSON and runs commands
- egress-filtered by nftables to an IP allowlist, deny by default
- killed after a maximum lifetime, with its namespace, chroot, and overlay removed
A warm pool keeps N restored VMs idle so create returns in about a millisecond. The pool is a buffered Go channel. When you take one out, a goroutine restores another.
The host
I did this on a Google Cloud n2-standard-4 with nested virtualization enabled: 4 vCPUs, 16 GB, Debian 13. Nested KVM adds overhead, so treat every number here as a ceiling. Bare metal would be faster. I chose it anyway because it costs twenty cents an hour and I live in India, where no provider I tried had bare metal in a nearby region.
The numbers
Guest: 1 vCPU, 512 MB, Debian bookworm rootfs, kernel 6.1, Firecracker 1.13.
| Metric | Value |
| --- | --- |
| Cold boot to agent answering over vsock | ~1.4 s |
| Snapshot restore to agent answering | p50 119 ms, min 81 ms |
| Exec round trip, warm sandbox, true | p50 4 ms, p99 9 ms |
| Idle RSS per restored VM | ~28 MB |
| Snapshot | 512 MB memory file, 15 KB state |
The restore number is the one vendors quote and it is real. A 512 MB guest comes back with its userspace intact in about a hundred milliseconds. The memory file is mmapped, so a restored VM that touches little memory costs the host 28 MB, not 512.
The 4 ms exec round trip surprised me. That is an HTTP request to the daemon, a Unix socket connect to Firecracker's vsock proxy, a handshake, a JSON message, fork and exec inside the guest, and the response back. Nested virtualization included.
The slowest thing in the system was cp
The first time I started the pool with 16 slots, create latency went from milliseconds to 33 seconds. The restore log said every VM was ready in 200 to 500 ms. Something else was eating the time.
It was the rootfs overlay. Each VM needs a writable copy of the 2 GB base image. On ext4, cp is a full copy, about 680 MB of real data per VM. Sixteen of those at once on a throughput-limited cloud disk queue behind each other:
| Host filesystem | Overlay copy per VM, 16 concurrent | Disk for 16 VMs | | --- | --- | --- | | ext4, full copy | 24 to 42 seconds | ~11 GB | | XFS with reflink | ~110 ms | ~100 MB |
I formatted a loop image as XFS with reflink=1, moved the state directory onto it, and the same cp --reflink=auto became a metadata operation. The restore path never changed. The vendor benchmark was accurate and irrelevant.
The lesson generalizes: measure from the API call to the first useful response, not from the point where the exciting technology starts. My "ready in" log line originally started its clock after the overlay was created. It was lying by omission.
What oversubscription does to exec latency
With reflink in place I ran a sweep: N sandboxes created at once, five commands each.
| Concurrent sandboxes | Pool size | Create p50 / p99 | Exec p50 / p99 | | --- | --- | --- | --- | | 5 | 4 | 2 ms / 148 ms | 41 ms / 251 ms | | 10 | 4 | 1.3 s / 2.7 s | 37 ms / 253 ms | | 20 | 4 | 3.0 s / 5.8 s | 40 ms / 227 ms | | 20 | 16 | 11 ms / 2.8 s | 173 ms / 1.4 s | | 40 | 16 | 2.4 s / 5.1 s | 206 ms / 1.8 s |
Two things fall out of this.
Create latency is bimodal. It is a pool hit at a millisecond or a pool miss at the refill rate, and refills serialize on CPU at 300 to 500 ms each under load. Pool size is your create SLO, and it costs 28 MB per slot. Size it for your burst, not your average.
Exec latency is about host cores. Twenty single-vCPU guests on four host cores is five-to-one oversubscription, and exec p99 went from 250 ms to 1.4 s. The rough rule from this box: one host core for every four or five mostly-idle sandboxes if you want exec p99 under half a second. Nested virt inflates this; I would expect metal to tolerate more.
Zero errors across every run. The thing is stable. It is just slower than you want if you starve it.
Fourteen things that broke
Firecracker booted on the first try. These are the things that did not, in the order they happened. Every one of them cost more time than the Firecracker part.
Building the rootfs. My first attempt ran cp -a / /out inside a container to dump its filesystem. /out is under /. It copies itself forever. Use docker export.
The kernel URL. Firecracker's CI bucket has vmlinux-6.1.141, not vmlinux-6.1.bin. My 314-byte kernel was an S3 error page.
nftables keywords. fwd, nat, and snat are reserved words. Naming a chain after one produces a syntax error that points at the wrong token. Prefix your chain names.
Interface name length. Linux caps interface names at 15 characters. My sb-<12 hex>h was 16, and the error was "Attribute failed policy validation," which tells you nothing. Shorter IDs.
cgroup v2. Debian 13 is v2 only. The jailer defaults to v1 and dies with CgroupHierarchyMissing. One config field.
The SDK links drives for you. The Go SDK's jailer support hardlinks every drive into the chroot. I had already placed the overlay there, so it failed with "file exists." Put the overlay one directory up and let the SDK move it.
The jailer drops privileges. Firecracker runs as an unprivileged user inside the jail. A root-owned overlay gave "Permission denied" when attaching the block device. chown it.
Go does not know vsock. net.FileListener on an AF_VSOCK socket fails with "address family not supported by protocol." The guest agent has to accept connections with raw unix.Accept and wrap the descriptor itself.
init has no PATH. The agent is PID 1. It starts with an empty environment. exec: "python3": executable file not found in $PATH. Set it yourself.
My firewall broke Docker. I gave my forward chain a default drop policy. That is the same hook Docker's containers go through. apt-get inside every container on the host hung for the rest of the evening until I noticed. Filter by ingress interface only.
Docker's firewall broke me. Docker sets its own FORWARD chain to drop. In nftables, a drop in any table wins. My allowlisted egress was silently blackholed. Add explicit accept rules for your bridge in Docker's chain.
Snapshots and the jailer do not compose in the SDK. Snapshot mode stats the memory file on the host, but the path is relative to the chroot. It re-adds the vsock device that the snapshot already restores. And it drops the handler that links the rootfs into the chroot. Three edits to the handler list after machine creation fix it. None of this is documented.
pkill matches its own sudo wrapper. sudo pkill sandboxd kills the sudo process whose command line contains "sandboxd" and exits before reaching the daemon. Use pkill -x or kill by PID.
Restored guests have a frozen clock. Every VM restored from my snapshot thinks it is 18:55 on the day I took it. TLS certificate validation will eventually notice. The fix is for the agent to set the clock from the host on first contact. I have not done it yet.
What I would tell someone starting this
Pick a reflink filesystem for the state directory before you measure anything. XFS or btrfs. It changes the shape of the whole system.
Instrument the entire create path from the first line. My worst hour came from a log line that started its timer too late.
Run it on a host with nothing else on it, or learn nftables well enough to coexist with Docker. Two of my fourteen failures were the two firewalls fighting.
Budget for the SDK. The Firecracker Go SDK is good for boot-from-scratch and thin for snapshots under the jailer. You will read its source.
And the actual Firecracker part really is the easy five percent. It booted a kernel in 1.4 seconds the first time I asked. Everything else is Linux.
What is next
Guest clock and DNS after restore. A rerun on bare metal for un-nested numbers. And a comparison against just paying E2B or Modal, which is the question this whole exercise exists to answer.