Spares You Haven't Touched Aren't Spares: Running a Quarterly Fire Drill on Idle Modem Gear

Every rack owner I've talked to has the same drawer. One modem, still in the anti-static bag it shipped in. A spare power brick with what looks like the right barrel jack. An antenna adapter that was the right gender eighteen months ago. And a mental note that says: if something dies, I'm covered.

That's not disaster recovery. That's a hypothesis you haven't tested.

A spare isn't hardware, it's a procedure

The failure mode is rarely "the spare won't power on." Modems are stubborn little things, and most of them survive a shelf just fine. What actually bites you at 2AM is the accumulation of small mismatches: firmware two releases behind the fleet, an IMEI the carrier portal has never seen, a SIM that got quietly suspended because it sat idle past the carrier's inactivity window, or a config file from before you moved the APN.

None of those show up when you plug the thing in and watch an LED blink. All of them show up when you're swapping under pressure with a customer on the phone.

So the audit I recommend isn't an inventory count. It's a fire drill — timed, documented, and repeated on a schedule.

Do the drill with a stopwatch, not a spreadsheet

Pick a quiet afternoon. Announce it to nobody, because it's just you. Then do this:

1. Unplug a live modem from the rack — ideally one carrying real traffic, or a canary line if you keep one.
2. Replace it with the cold spare, following only the written procedure.
3. Time it, from unplug to traffic flowing again on the spare.
4. Write down every moment you had to guess.

That last step is the whole point. The guesses are the bugs in your runbook. If you had to remember which way the SIM tray faces, or dig for a config, or log into the carrier portal to re-authorize an IMEI, that's a gap — and at 2AM with cold hands you'll hit it twice as slowly.

When I last ran this on my own rack, the swap took eleven minutes. Eight of those were spent finding the right barrel adapter. The procedure said "connect power." It did not say which of four bricks in the drawer was the 12V/2A unit with the 5.5x2.1mm tip rather than the 5.5x2.5mm one. That's a real gap, and it's exactly the kind that turns a fifteen-minute outage into a ninety-minute one.

The four checks that fail most often

Firmware drift. Your production fleet probably got updated at some point through whatever management tool you use. The spare didn't, because it wasn't plugged in. Pin the spare to the fleet's version, or at least confirm you can bring it current with tooling you still have. If the only way to flash it is a vendor utility you last ran in 2023, test that utility too — on the same laptop you'd grab in a hurry.

IMEI registration. On US carriers, an IMEI that isn't on the account or in the right device pool may register on the network and then refuse data, get throttled, or trip a fraud flag. Whatever your carrier's process is — portal entry, API, a support ticket — the spare's IMEI needs to already be in it, or you need a documented one-step way to add it. "I'll call support" is not a step at 2AM.

SIM state. This is the sneaky one. Many US carriers will suspend or recycle a line that hasn't shown activity for some number of months. The exact window varies by carrier, plan, and whatever the terms said last quarter, so don't trust a figure you read on a forum years ago — check your carrier's current language. What you can trust: a suspended SIM looks perfectly healthy sitting in a slot. The only way to know is to push a test message or hit a data endpoint on a schedule. Put the spare SIM on the same canary cadence as everything else.

Physical compatibility. Modem to hub, hub to host, host to power, antenna to connector. Gender changers, RP-SMA versus SMA, the hub's per-port current budget, whether the spare needs a powered hub while the production unit doesn't. And: does the spare even fit? I've seen a "drop-in replacement" that was four millimeters too tall for the enclosure lid.

Write the swap card, not the swap doc

Nobody reads a twelve-page runbook at 2AM, including the person who wrote it. What works is one index card, or one pinned note, in order:

• Stop the service. The exact command.
• Which port and which cable to pull.
• Where the spare lives, and what its label says.
• The config path and the one-line command that applies it.
• The IMEI, the SIM ICCID, and the APN.
• How to verify traffic is actually flowing (the real test, not "check the dashboard").
• Who to call if it doesn't come up, with a phone number.

Laminate it or pin it. Date it. Update it every time the drill proves it wrong — which it will, the first time.

Put a name and a date on it

Quarterly is enough. Put it on the calendar with your name next to it, the way you'd book a dentist appointment, because the alternative is doing it "when things calm down," which is never.

The drill costs about an hour including cleanup. The alternative — discovering at 2AM that your spare has a suspended SIM and firmware from the previous administration — costs considerably more, and it happens in front of a customer.

Your spare isn't a spare until you've proved it. Until then it's a box on a shelf and a story you tell yourself.