I Fixed a Shutdown Race Condition That Could Leave Bot Processes Running Forever

#devchallenge #bugsmash

Summer Bug Smash: Clear the Lineup 🐛🛹

This is a submission for *DEV's Summer Bug Smash: Clear the Lineup** powered by Sentry.*

Fixing a Shutdown Race Condition in StayPresent

Project Overview

I maintain StayPresent, an open-source Python library that helps developers keep bots and background services running reliably on platforms like Render, Railway, Koyeb, and Heroku.

StayPresent launches a lightweight web server alongside one or more bot processes, monitors them, and automatically restarts them if they crash. Since it's designed for long-running services, reliability is the highest priority—even rare edge cases can become real production problems over time.

While preparing v1.5.10, I discovered a subtle race condition that could interfere with graceful shutdown under a very specific sequence of events.

The Bug

When a supervised bot crashes, StayPresent waits for a configurable restart delay before launching a replacement process.

That restart delay originally used an uninterruptible time.sleep().

Most of the time this worked perfectly. The problem appeared if the application received Ctrl+C or SIGTERM during that restart window.

The sequence looked like this:

A bot crashes.
The monitor thread begins waiting before restarting it.
Shutdown starts because the user presses Ctrl+C or the platform sends SIGTERM.
Shutdown terminates every process it knows about.
The sleeping monitor thread wakes up and launches a brand-new bot after shutdown has already finished cleaning up.

Because monitor threads are intentionally non-daemon, the application could then wait indefinitely for a process that should never have existed.

The race window was small, but it affected one of the most important guarantees a long-running service should provide: when shutdown begins, no new work should start.

The Fix

The solution focused on making shutdown authoritative.

I made two changes:

Replaced the restart delay with an interruptible wait so pending restarts immediately abort when shutdown begins.
Synchronized the restart path with the shutdown sequence so a monitor thread can never spawn a new process after shutdown has started.

Together, these changes eliminate the race entirely while preserving the existing restart behavior during normal operation.

The public API didn't change, and users don't need to modify their code—the improvement is entirely internal.

Verification

Concurrency bugs are notoriously difficult to reproduce because they depend on timing.

To gain confidence in the fix, I repeatedly tested scenarios including:

Bots crashing immediately before shutdown.
Ctrl+C during restart backoff.
SIGTERM while supervising multiple bots.
Repeated crash/restart cycles.
Normal graceful shutdown with healthy processes.

Across all of these cases, shutdown consistently took priority over restarting processes, and no orphaned bot could be created once termination had begun.

Why This Bug Matters

This wasn't a bug most users would encounter during everyday development.

It only appeared when process crashes, restart delays, signal handling, and thread scheduling aligned in exactly the wrong order.

Those are often the hardest bugs to find because they hide in the interactions between otherwise correct pieces of code.

For infrastructure libraries like StayPresent, however, these edge cases matter. Users trust the library to supervise their applications reliably, and that includes shutting them down cleanly under every circumstance.

Fixing this race condition makes that guarantee significantly stronger.

Conclusion

One of my favorite aspects of systems programming is that the hardest bugs are rarely syntax errors—they're usually subtle interactions between threads, processes, and operating-system signals.

This fix didn't add a new feature or change the API.

Instead, it strengthened one of the library's core reliability guarantees: once shutdown begins, shutdown always wins.

Sometimes the most valuable improvements are the ones users never notice—because everything simply works.