On Sunday 11 October at 21:53 Malaysia time, all of BoxCompute went down: every VM, the API that starts them, and the web app. It stayed down for 2 hours 41 minutes. Bringing it back took a drive to the data center, a power cycle, and a few services started by hand.
This is what failed, and what we changed.
One port on one server
BoxCompute runs on our own servers in a colocation facility in Cyberjaya. One of them, sandbox-01, is a Lenovo ThinkStation P700. On 11 October it ran every customer VM, the sandbox API, and the web app's only database.
It has two network ports on the board, and only one was cabled: an Intel I218-LM on Linux's e1000e driver. At 21:53:37 the kernel started logging this:
e1000e 0000:00:19.0 lan0: Detected Hardware Unit Hang
It logged that line every two seconds, 4,567 times. Linux kept running, but the port sent nothing, so to everything else the server was gone. The web app still loaded, but every request that needed the database failed, sign-in included.
Our alert for a server losing its internet connection fired about eight minutes in. It said the connection was lost, but not which server. Our public health check stayed green the whole time, because the endpoint it calls never touches the database. And our first announcement, at 22:40, said the web app was still up. It wasn't.
Why I had to drive
The hung port was the server's only connection, so nothing could reach it. At 00:25 I powered it off at the rack and on again, and two minutes later it was back on our private network.
It still wasn't serving anything.
The reboot we had never tested
k3s, the Kubernetes that runs everything on that machine, hadn't started, and nothing said why.
The boot log had the answer: systemd had found ordering cycles. The disk mounts k3s needs were set to come up before the local filesystems were ready, while also waiting on a service that could only start after them. systemd broke each loop by dropping jobs, and k3s, waiting on those mounts, never started.
That loop had been there since 11 September. Every change since then was applied to a running machine, where the filesystems were already up, so nothing hit it. Only a cold boot does, and we had never tested one with those units.
I started the mounts and k3s by hand, in order. At 00:32 the server was ready, and at 00:33 the web app could reach its database again. Before anything else, I copied the database off the machine and checked the copy.
Then I moved the cable. The old port worked again after the restart, but we moved off it anyway: at 00:46 the cable went into the second port, an Intel I210 on a different driver (igb). The server was offline for about three seconds.
We still don't know why the port hung. The logs showed no flood of connections and no rise in log volume before it, but we weren't recording network traffic then, so we can't rule traffic out.
What was lost
No customer files: every VM's disk was kept. One VM that was running when the server dropped off was marked finished instead of asleep, so its owner couldn't start it again even though its disk was intact. The controller now puts a VM to sleep when the machine under it fails, so it can be started again.
What we changed
- Alerts say what broke and where. A server that stops reporting, a network port with errors, and unusual network traffic each have their own alert, and every alert names the server. Every server's network traffic is now recorded.
- Backups leave the building. Both databases were already copied every hour, but the copies stayed in the same data center. Since 12 October they are also copied to S3 every six hours, each copy is restored to check it, and an alert fires if the copies go stale.
Still to do
- Fix the boot, and watch the port. A fix for the startup loop is written, along with a watchdog that resets the network driver if the port hangs again. The watchdog never reboots the server. Turning them on needs one reboot with someone at the rack, about ten minutes of downtime, which we'll announce first.
- Move the database. The web app's database still runs on sandbox-01. A replica on another server, or moving it off the VM host, is next.
- Check the database from outside. The public health check should fail when the database does.
One port on one server could take everything down, and the way back had never been tried. The second part is the one I'd pass on: if a machine hasn't been cold-booted since its last config change, you don't know that it boots.