I Almost Shipped a Production Outage Twice — Because the Dashboard Said Everything Was Fine

The model, briefly

A tiny shared loader reads a layered set of `.env.<stage>` files into `process.env`. A typed registry marks each variable public, secret, or connection — only the public ones cross to the browser, snapshotted into a context provider at request time. No secrets in the bundle, and rotating a URL means restarting a process, not rebuilding an image. The migration itself matched that description exactly: grep for the old prefix, drive it to zero, verify a clean build. Nothing about it hinted at what was coming.

Lie #1: “decrypt” didn’t mean decrypted

Moving a value between two environment variables on the hosting platform, I called its API with `decrypt=true` and copied the returned value into the new key. The frontend that read it 500’d on every route with “Invalid URL.”

The returned “decrypted” value wasn’t decrypted. My token didn’t have decrypt rights, so the API quietly handed back the ciphertext blob instead — and I’d written a thousand-character base64 string into the variable as if it were a URL. At runtime, the client library got that blob, tried to parse it as a URL, and threw.

I burned two wrong hypotheses guessing at this — edge-runtime env access, dynamic versus static lookups. What broke the loop was refusing to guess a third time: I shipped a throwaway diagnostic that returned the actual error and a few booleans in response headers. One request later, the real cause was obvious. Lesson: after any API-driven secret copy, read the value back and assert its shape. And once you’re two guesses deep, stop guessing and instrument.

Lie #2: a “rollback” quietly became a pin

Mid-incident, I rolled the two broken frontends back to their last-good deploy to restore service. Correct move. What I didn’t know: this platform’s instant rollback pins the production domain to that specific deployment. Afterward, my fixed builds went green — READY, healthy, everything — but the production domain kept serving the old, rolled-back build. New deploys no longer auto-took the domain.

That’s the nasty part: both the old and the new code returned 200 on the page I was checking, so status alone couldn’t tell me which one was actually live. I briefly declared an app “fixed in prod” while it was still serving the rollback. Lesson: after a rollback, you have to explicitly promote the next build, and verify the domain’s alias points at the specific deployment you think it does — not just that the URL returns 200.

Two supporting traps

• A monorepo’s per-app build doesn’t build your workspace libraries. An app importing a sibling library through its built dist/ output fails with “module not found” on a clean CI build, even though it works locally — because your local tree still has stale dist/ from an earlier full build. Wipe everything and build clean to reproduce it; fix by building the libs first. “Works on my machine” is often just “stale artifacts on my machine.”
• Edge middleware runs on every route and reads env differently. It wants static process.env.NAME references, not a dynamic lookup helper, and an error there takes down the whole app, not one page. Keep the heavy config machinery out of the edge bundle.

What to trust instead of the dashboard

Every one of these was invisible at build time and green in the platform dashboard. The deploy was “successful” while the app was down or stale, twice, for two completely different reasons. The only signals that turned out to be reliable were the ones I generated myself against the live surface: the actual runtime error instead of the status code, the real alias target instead of the “rolled back” label, a from-scratch clean build instead of a green CI run. A platform’s checkmark tells you the platform did its job. It doesn’t tell you your app is actually up — and that’s the same discipline I run on every deploy I ship, including yours: never take the platform’s word for it when the real thing can be checked in seconds.