The Hidden Second Message: Auditing SMS Encoding Before It Hits Your Invoice

Most SMS billing surprises do not begin with a carrier rate hike. They begin with a character that looks harmless in a web form but forces the gateway to abandon the compact GSM alphabet and switch to UCS-2. On a busy modem rack, that switch can turn one billable message into two, three, or more segments without anyone touching the price list.

The frustrating part is that the message still looks like one text on the handset. Your application sees one string. Your customer sees one bubble. The carrier sees a different segment count, and eventually so does your invoice. If you run SMS infrastructure for a living, encoding is not a developer detail. It is a billing control.

Why 120 characters can bill as two

GSM 7-bit is the lean encoding most US SMS traffic uses when the content stays inside the GSM 03.38 alphabet. A single message gets 160 septets. Once you add a user data header for concatenation, the practical payload drops to 153 characters per segment. That is well known. What gets missed is how easily the encoding changes.

Add one emoji, one curly apostrophe copied from a word processor, or one accented character outside the GSM alphabet, and many gateways switch the entire payload to UCS-2. Now the single-message limit is 70 UTF-16 code units, and concatenated segments top out around 67. An emoji outside the Basic Multilingual Plane, such as many common smileys, consumes two UTF-16 code units by itself. A 158-character note with one emoji at the end can leap from one segment to three.

There is a second trap: GSM 7-bit extension characters. Brackets, braces, the pipe symbol, the caret, the tilde, and the euro sign are represented as escape sequences. They count as two septets each. A message full of square brackets can hit the segment boundary earlier than a character counter suggests. Neither your JavaScript string length nor a naive Python len call necessarily matches what the carrier bills.

The three encodings that matter in practice

On a US modem farm, you mostly care about three outcomes. First, pure GSM 7-bit: cheap, compact, and ideal for alerts and transactional text. Second, UCS-2: necessary for emoji, many non-Latin scripts, and typographic punctuation, but expensive because each character eats more payload. Third, concatenated messages in either encoding: still one user experience, but multiple billable segments.

The audit question is not which encoding is best in theory. It is which encoding your gateway actually selected, which one your logs claim, and which one the carrier charged. Those three answers drift apart after firmware updates, library upgrades, and copy changes. A message template that tested clean in staging can start flipping to UCS-2 when someone replaces a straight apostrophe with a typographic one.

Build a canary set before the invoice arrives

You do not need a lab full of handsets to catch this. You need a repeatable test set and a way to compare gateway logs with carrier records. Send the same six messages through every path your production traffic uses: application to gateway API, gateway to SMPP bind, and gateway to modem. Include at least one route through each carrier you rely on.

The canary set should include a 160-character plain GSM message, a 161-character plain message, a 159-character message with one emoji at the end, a 70-character message containing an accented character, a message packed with GSM extension characters, and a long message that must concatenate. Keep the text fixed. Do not let marketing rewrite the canary. The point is to compare segment counts over time, not to test new copy.

For each send, record what the gateway reports for encoding and segments, what the modem actually transmits, and what appears in the carrier or aggregator detail record. If the gateway says one segment and the carrier bills three, you have found a real cost leak. If both say three but your application expected one, you have found a product or copy problem.

Log the fields your invoice cares about

Most SMS logs are built for debugging delivery, not for finance. Add encoding-aware fields at the gateway boundary: requested encoding, detected encoding, GSM septet count, UTF-16 code unit count, user data header bytes, calculated segment count, and billed segment count when available. Store the message body hash so you can compare test runs without retaining sensitive content.

Then alert on mismatches. A sudden rise in UCS-2 share is a useful signal. So is a jump in average segments per message for a template that used to be stable. If your gateway exposes the SMPP data_coding value, log it. A gateway that silently changes data_coding from GSM 7-bit to UCS-2 is doing exactly what it was configured to do, but someone still needs to own that decision.

Fix defaults before they fix your budget

Start with normalization. Convert curly quotes to straight quotes, en dashes to hyphens, and other typographic characters to GSM-safe equivalents when the content allows it. Strip emoji from alerts that do not need them. For templates that genuinely require Unicode, accept the cost but set expectations in code review.

Next, replace naive length checks with a segment calculator that understands GSM 7-bit, extension characters, UCS-2, and concatenation headers. Enforce maximum lengths per encoding at the application layer, not only at the gateway. Finally, confirm that your gateway and modem firmware agree on encoding defaults. A modem rack with mixed firmware versions is a fine place for encoding assumptions to rot.

Put the audit on a calendar

Run the canary set monthly, after every gateway upgrade, after any carrier or aggregator change, and whenever a high-volume template gets edited. Keep the results in a simple table. The goal is not perfection. The goal is to notice the day a single emoji starts costing you a second segment across a hundred thousand messages.

Encoding audits are boring until they are expensive. A few fixed test messages and a handful of log fields will catch more invoice surprises than another rate negotiation. On a modem farm, the cheapest message is the one that stays in the alphabet you thought you were using.