Your Modem Farm Has a Version Control Problem That Git Can't Fix

Two modems arrive on the same purchase order. Same model, same batch sticker, same vendor. Eleven months later, one is a workhorse and the other drops its PDP context every couple of days and sometimes just refuses to register after a reboot. You swap SIMs, swap antennas, swap power supplies, reseat the USB hub — nothing moves the needle. Then you plug both into a laptop, run AT+CGMR, and the version strings don't match. That's the moment the mystery stops being a mystery and becomes a maintenance problem.

Firmware drift is boring in the way that a slow leak is boring, right up until the ceiling comes down. This is a walkthrough of how to audit it across a live GSM rack, what to actually compare, and how to decide whether a mixed-firmware farm is worth the effort to homogenize.

Drift Doesn't Announce Itself

Nobody plans to run six firmware versions on twelve modems. It happens in pieces. A vendor ships a batch with a newer build than the last order. A carrier push lands on some devices and not others because half the SIMs were sitting in a quarantine bin that week. Someone flashes a modem at 11 p.m. trying to fix a registration issue, gets it working, and forgets to write it down. A replacement unit arrives from RMA with whatever the refurb line happened to load.

Because the devices all look identical from the outside — same LEDs, same USB descriptor, same model silkscreen — the drift stays hidden until behavior diverges. And by then you're debugging symptoms, not causes.

Step One: Make the Invisible Visible

You can't audit what you haven't inventoried. Build a spreadsheet or a small database with a row per modem and columns for serial number, IMEI, model, firmware build string, carrier profile, flash date, flash source (factory, OTA, manual, vendor tool), and a free-text notes field. It feels like busywork for the first twenty rows. It pays for itself the first time a modem acts up.

On Linux, the quick-and-dirty pull is straightforward. For serial-attached modems, open the AT port and send AT+CGMR for the firmware revision and ATI for model info. For ModemManager-managed devices, mmcli -m 0 --output-keyvalue gives you firmware revision, hardware revision, and the current operator in one shot. If you're running a SIM bank behind an aggregation layer, many controllers expose the same fields over their own API — grab them there too so the inventory isn't split across two systems.

Automate the collection if you have more than a handful of modems. A short Python script looping over /dev/ttyUSB* with pyserial will do it. Dump the results to CSV, commit the CSV to git, and now you have a history. The git part matters more than it sounds — version drift is a timeline, and a timeline without commits is just a guess.

Version Strings Are Not Behavior

Here's where people stop too early. Matching firmware strings across the farm is necessary but not sufficient. Two modems on the same build can still behave differently if the carrier profile on the SIM side is stale, or if one was flashed with a generic image and the other with a carrier-approved one.

So pair the version audit with a light behavioral baseline. Once a quarter, run the same small suite against every modem: time to register after cold boot, a short throughput sample, an SMS round-trip, and a reboot-recovery check. Keep the numbers. You're not looking for perfection — you're looking for outliers relative to the fleet's own baseline. A modem that takes ninety seconds to register when the median is twelve is telling you something, and it's usually telling you before it fails outright.

The Carrier-Approval Trap

In the US, the big three certify specific firmware builds against their networks. A build that shipped generic from the factory may work fine on T-Mobile today and fall over after a network-side change three months from now, because it was never in the certified set to begin with. This is the mismatch that bites operators hardest, because the failure often arrives weeks after the flash and looks like a hardware problem.

Check the firmware string against the carrier's published approved list whenever you have one. If you can't find a list, ask your account rep — for a business account they usually have one. Record the result in your inventory. Models marked unapproved aren't necessarily broken, but they belong in a different risk bucket, and you should know which bucket they're in before a customer does.

Test Rollback Before You Need It

Every firmware audit should answer one question: if this update goes sideways, how do we get back? That means keeping a known-good image per model, confirming your flash tool actually works on a spare unit, and knowing whether the bootloader is locked. Plenty of operators have discovered, mid-incident, that the rollback path they assumed existed was never there.

Do this on a Tuesday morning, not a Friday night. Flash a spare to the current build, then flash it back, and time both. Write down the procedure. The point isn't paranoia; it's that rollback capability is the thing that makes every other firmware decision reversible.

When Mixed Firmware Is Fine — and When It Isn't

A mixed farm isn't automatically a disaster. If you're running a lab rack, a handful of spares, or a low-volume verification setup, drift is annoying but tolerable. The calculus changes when SIMs are generating revenue, when SMS deliverability is on the line, or when support burden is measured in hours per week.

A rough rule: the more your rack is a production asset, the more homogeneity is worth paying for. Standardize on one build per model, flash the stragglers during a maintenance window, and freeze the fleet until the next planned update. Keep the spares imaged to match. The goal isn't perfect uniformity forever — it's knowing exactly what you're running and why, so the next weird failure doesn't send you down a three-day rabbit hole.

Put It on the Calendar

Run the inventory sweep quarterly, plus any time a carrier announces a network change or a vendor pushes an update. Thirty minutes of scripted collection and a ten-minute review beats a weekend of debugging. Your rack will keep running either way — the question is whether you find out about drift from a dashboard or from an angry customer.