DEV Community

ricco020
ricco020

Posted on

A failed deployment is invisible from the web

#devops #monitoring #webdev #typescript

Last night a deployment of mine failed for 21 hours and every monitoring check stayed green. The site answered 200 the whole time. Here is why that happens, and the two-line check that catches it.

The failure mode

When a build fails on Vercel (and on Railway, and on most platforms with atomic deploys), the platform does the sensible thing: it keeps serving the previous successful build. Your users see a working site. Your uptime monitor sees 200 OK. Your health check passes.

What nobody sees is that the commit you pushed is not the code being served. Every article, fix and config change since the last green build is sitting in git, live nowhere.

That is a good default — far better than serving a broken build. But it means "the site is up" and "my work is deployed" are two different questions, and most monitoring only answers the first.

The check that actually answers it

Compare the timestamp of your last commit with the timestamp of the last successful deployment. If the commit is newer by more than a build's duration, something is wrong:

git log -1 --format=%ct                 # last commit, unix time
npx vercel ls --prod                    # last deployments + their state
Enter fullscreen mode Exit fullscreen mode

If the age of the newest Ready deployment is older than your last commit, you are serving stale code. In my case the newest Ready was 21 hours old, and two Error deployments sat above it.

The trap that cost me an hour

Here is the part worth the read. Outside a TTY, vercel ls splits its output across two streams:

  • stdout gets the deployment URLs, one per line — no status
  • stderr gets the formatted table, including the ● Ready / ● Error column

So this looks correct and silently returns nothing:

r = subprocess.run(["npx","vercel","ls","--prod"], capture_output=True, text=True)
errors = [l for l in r.stdout.splitlines() if "Error" in l]   # always empty
Enter fullscreen mode Exit fullscreen mode

Your script reports zero errors on a project that is entirely broken. The fix is to read both streams:

out = (r.stdout or "") + (r.stderr or "")
Enter fullscreen mode Exit fullscreen mode

The general lesson is worth more than the specific fix: an empty result and a clean result are not the same thing. If a check can return "nothing found" both when everything is fine and when the parsing broke, it will eventually lie to you — and it will lie in the reassuring direction. Make the two cases distinguishable:

if not lines:
    return "NOT VERIFIED"     # not "0 errors"
Enter fullscreen mode Exit fullscreen mode

What actually broke the build

For the curious: a TypeScript error on a component prop. One call site out of 21 was missing a required id. Nothing exotic — but because the previous build kept serving, nothing surfaced it either.

I found the second occurrence only after the first fix failed, because I had listed the call sites with grep ... | head -8 on a file with 21 of them. A head on a search meant to be exhaustive turns "I saw nothing" into "there is nothing".

Three things to take away

  1. Uptime monitoring does not detect failed deployments. Compare commit time to last successful deploy time.
  2. Check the state of recent deployments, not just the age of the last good one.
  3. When a check returns nothing, make sure it can tell you why — no data and no problem must not look identical.

If you are wiring up this kind of monitoring, the same reasoning applies to the telemetry you collect about it — what OpenTelemetry actually is, and what it deliberately does not give you covers the difference between producing telemetry and having observability, which is the trap one layer up.

Top comments (0)