The security emails had never sent once, and nothing anywhere complained
August 6, 2026
Several product apps I work on share one small internal service: it sends the “we noticed a new sign-in on your account” emails. Each app signs its request with its own secret key, the service verifies that signature, and that signature is the only thing standing between the service and being an open relay for anyone who finds the URL. I went in to add replay protection to it. What I found first was that the whole feature had never worked — not once, in production, since the day it shipped.
Zero secrets on one side, a signature on every request from the other
The service verifies each incoming request against its configured signing secrets. There were none. Meanwhile every calling app was dutifully computing a signature and attaching it to every send. So every request arrived, got checked against an empty set, and came back 401 Unauthorized. Every single one, for the entire life of the feature.
Nobody noticed, and the reason nobody noticed is the part worth sitting with. Sending a security notification must never block someone from logging in — if the email service is slow or down, you let the user in anyway and move on. So the call is deliberately fire-and-forget: kick it off, don’t await it, ignore the result. That is the correct design. It also means a permanent, total failure looks exactly like a healthy system from every angle a person normally checks. No error surfaced in the app. No support ticket, because nobody complains about an email they never knew they were supposed to get.
A feature that fails loudly gets fixed the same afternoon. A feature that fails silently can stay broken for its entire life.
Proving the keys matched without emailing real people
Setting fresh secrets on both sides is a five-minute job. Proving they actually match is the hard part, because the obvious test is to send a real email — to a real person, from production, repeatedly, while you fiddle with keys.
The endpoint handed me a free way out, and it was an accident of ordering: it checks the signature before it checks whether the event type is one it is allowed to send. So a request carrying a valid signature and a deliberately bogus event type comes back 403 event_not_allowed. That response is proof the secret matched. A wrong secret returns 401 instead. Two different failures, two different meanings, and neither one sends mail.
One probe per app, zero emails, and unambiguous evidence that each key pair lines up. I now look for that shape on purpose: some way to exercise an authentication path end to end without triggering the side effect sitting behind it. Where one doesn’t exist by accident, it is usually worth building.
The thing I actually came to fix: replays
The original signature covered the request body and nothing else. A request captured once could be replayed forever — the only thing resembling a limit was an in-memory, per-instance rate limiter, which across a serverless fleet is no limit at all.
The fix is standard: bind the signature to time. The caller now sends a timestamp header and signs the timestamp together with the body, and the service rejects anything outside a five-minute window. A captured request stops working the moment the window closes.
The interesting part is shipping that when every caller is already live and you cannot flip both sides at the same instant. So the verifier learned to accept both schemes at once — the new signature when a timestamp header is present, the legacy one otherwise — with a single flag deciding whether legacy is still allowed. That turns the rollout into an ordered sequence where every step is independently reversible:
- Deploy the verifier that accepts both schemes.
- Publish the updated client library.
- Redeploy each app so it starts signing with a timestamp.
- Flip the strict flag, and legacy signatures stop being accepted.
While I was in there I also made the verifier take a list of candidate secrets rather than a single one, so rotating a key becomes an overlap window — accept old and new for a while — instead of a synchronized cutover where somebody has to be wrong for a few seconds.
Three things that bit me
- Write-only secrets are genuinely write-only. The platform’s “sensitive” environment variables cannot be read back, so I could not harvest the existing values to copy them across. If you cannot read a secret, you cannot migrate it — regenerate it and set both sides.
- Deploy production from a clean, committed tree. A CLI deploy run from the wrong working directory shipped uncommitted local files and dropped a config file, quietly changing routing. Deploy from a known-good commit, then verify the deployed artifact rather than the source you believe you deployed.
- Use the release tooling the repo already has. I hand-edited a version number and published, bypassing the repo’s changeset flow — which also skips the changelog. The clean fix was recording the changeset after the fact so the next version cut captured it properly.
The takeaway
The failures that survive longest are the ones designed not to interrupt anyone. If a code path is fire-and-forget by design — and a security email should be — nothing about normal operation will ever tell you it stopped working; you have to go ask it directly, and you need a way to ask that doesn’t page a real customer.
Capabilities this touches