Health Checks: Verifying Application and Dependency Health

A practical guide to health checks — the endpoints and mechanisms that let infrastructure verify whether an application (and its dependencies) are actually working — covering liveness vs. readiness vs. startup checks, ASP.NET Core's health check framework, dependency health checks, how orchestration platforms consume them, and common anti-patterns that make health checks actively harmful.

Table of Contents

1. Introduction
2. Why Health Checks Exist
3. Liveness, Readiness, and Startup: Three Different Questions
4. ASP.NET Core's Health Check Framework
5. Dependency Health Checks
6. Custom Health Checks
7. How Kubernetes Consumes Health Checks
8. How Load Balancers and Cloud Platforms Consume Health Checks
9. Health Checks for Background Services
10. Health Checks vs. Deep Monitoring
11. Designing Health Checks That Don't Cause Outages
12. Common Pitfalls
13. Quick Reference Table
14. Conclusion

Introduction

A health check is a deliberately simple endpoint or mechanism that answers one narrow question: is this specific application instance working right now, well enough to keep receiving traffic (or, in some cases, well enough to keep existing at all)? It sounds almost too simple to deserve a dedicated guide — but health checks sit at the exact intersection of nearly every infrastructure guide in this series (Kubernetes/Helm's probes, Docker's HEALTHCHECK, the Azure/AWS compute guides' load balancer integration, Background Services' worker monitoring), and getting them subtly wrong is a genuinely common cause of real production outages, not a purely theoretical risk.

app.MapHealthChecks("/health/live");
app.MapHealthChecks("/health/ready");

Two endpoints, a handful of lines of configuration — and yet the specific design decisions behind what each one actually checks, and how orchestration platforms interpret their responses, determine whether a struggling instance gets gracefully removed from rotation or whether a routine deployment turns into a cascading outage.

1. Why Health Checks Exist

The problem: "the process is running" isn't the same as "this instance can actually serve traffic"

A process can be technically alive — accepting TCP connections, responding to a basic ping — while being completely unable to do meaningful work: its database connection pool might be exhausted, a critical downstream dependency might be unreachable, or it might still be warming up caches after a fresh restart. Infrastructure making traffic-routing and restart decisions needs a more nuanced signal than "is the process running," which is exactly what a well-designed health check provides.

What health checks let infrastructure decide automatically

• Should a load balancer send traffic to this instance? (readiness, Section 2)
• Should an orchestrator restart this instance because it's stuck? (liveness, Section 2)
• Has this instance finished starting up enough to be evaluated normally yet? (startup, Section 2)
• Is this specific deployment/rollout actually succeeding, or should it be rolled back? (connecting directly to this series' CI/CD Pipelines and Kubernetes/Helm guides' deployment strategy discussions)

Health checks as the trust boundary between an application and its orchestrator

Every automated deployment and scaling decision covered elsewhere in this series — rolling updates (Kubernetes/Helm guide), deployment slots (Azure Compute guide), auto-scaling (Cloud Cost Optimization guide) — ultimately depends on the orchestrator being able to trust an application's own self-reported health signal. A health check that lies (reporting healthy when it isn't, or vice versa) doesn't just produce a wrong dashboard reading — it actively misleads the systems making real traffic and lifecycle decisions on the application's behalf.

2. Liveness, Readiness, and Startup: Three Different Questions

This is the single most important conceptual distinction in this entire guide, and conflating these three checks is the most common, most consequential health-check mistake in production systems.

Liveness: "Should this instance be restarted?"

Question: Is this process in a state so broken that killing and restarting it is the right fix?
Consequence of failure: the orchestrator KILLS and RESTARTS the instance

A liveness check should fail only when the application is in a state a restart would actually fix — a genuine deadlock, an unrecoverable internal state, a hung thread pool. It should emphatically not fail just because a downstream dependency (a database, a third-party API) is temporarily unavailable, since restarting the application does nothing to fix a downstream outage and instead adds unnecessary restart churn on top of an already-degraded situation.

Readiness: "Should this instance receive traffic right now?"

Question: Is this instance currently capable of successfully handling a request?
Consequence of failure: the orchestrator STOPS ROUTING TRAFFIC to this instance, but does NOT restart it

A readiness check is the right place to check dependency health (Section 4) — if the database is unreachable, this specific instance genuinely can't serve most requests successfully right now, so it's correct to stop sending it traffic. Critically, failing readiness doesn't restart the instance — it simply waits, and the instance automatically becomes eligible for traffic again once its readiness check starts passing, without ever needing a restart at all.

Startup: "Has this instance finished its initial warm-up?"

Question: Has this instance completed its (potentially slow) initialization yet?
Consequence of failure: the orchestrator waits longer before evaluating liveness/readiness at all

A startup probe (a distinct concept in Kubernetes specifically, Section 6) exists for applications with a genuinely slow startup sequence — loading a large cache, running warm-up queries — giving that slow startup a generous grace period without needing to set an equally generous, and therefore less useful, timeout on the liveness check that governs steady-state operation.

Why conflating these three causes real outages

// ❌ A dangerous, common mistake: the SAME check used for both liveness and readiness
app.MapHealthChecks("/health"); // includes a database connectivity check

// If the database has a brief outage:
// - readiness correctly fails → traffic stops routing here (fine, this is what should happen)
// - liveness ALSO fails (it's the same endpoint) → Kubernetes RESTARTS every instance
// - a brief database blip has now caused a full application restart storm, making recovery SLOWER

This exact scenario — a database check included in a liveness probe — is one of the most common real-world causes of a minor downstream issue escalating into a full application outage: instead of gracefully waiting out a temporary database blip (which readiness alone would handle correctly), every instance gets killed and restarted simultaneously, and if the database is still recovering when they all try to reconnect at once, the restart storm can actually make the underlying problem worse.

3. ASP.NET Core's Health Check Framework

Basic setup

builder.Services.AddHealthChecks();

var app = builder.Build();
app.MapHealthChecks("/health");

ASP.NET Core's built-in health check middleware (Microsoft.Extensions.Diagnostics.HealthChecks) provides the foundational framework — registering health check implementations, running them, and aggregating their results into an overall status, exposed via one or more mapped endpoints.

Separate endpoints for liveness and readiness

builder.Services.AddHealthChecks()
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: new[] { "live" })
    .AddSqlServer(connectionString, tags: new[] { "ready" })
    .AddRedis(redisConnectionString, tags: new[] { "ready" });

app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("live")
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready")
});

Tags are the mechanism for implementing the liveness/readiness distinction from Section 2 within a single health check registration — the live endpoint runs only the minimal, restart-worthy checks, while ready runs the full set including dependency checks, and each endpoint's Predicate filters which registered checks actually execute for that specific request.

Response formatting

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready"),
    ResponseWriter = async (context, report) =>
    {
        context.Response.ContentType = "application/json";
        var result = JsonSerializer.Serialize(new
        {
            status = report.Status.ToString(),
            checks = report.Entries.Select(e => new { name = e.Key, status = e.Value.Status.ToString(), description = e.Value.Description })
        });
        await context.Response.WriteAsync(result);
    }
});

The default response is minimal (just an HTTP status code and a plain-text status word), which is entirely sufficient for most orchestrators (they only care about the HTTP status code, Section 6) — a richer JSON response, as shown above, is more useful for human debugging (hitting the endpoint directly to see exactly which dependency is failing) without changing what the orchestrator itself actually consumes.

4. Dependency Health Checks

Database connectivity

builder.Services.AddHealthChecks()
    .AddSqlServer(connectionString, name: "sql-server", tags: new[] { "ready" })
    .AddNpgSql(postgresConnectionString, name: "postgresql", tags: new[] { "ready" });

Community-maintained health check packages (AspNetCore.HealthChecks.SqlServer, .Npgsql, .Redis, and many others covering most databases and infrastructure covered throughout this series) provide ready-made checks that verify actual connectivity — typically a lightweight query or ping — rather than requiring hand-written connection logic for every dependency type.

Message broker connectivity

builder.Services.AddHealthChecks()
    .AddRabbitMQ(rabbitConnectionString, tags: new[] { "ready" })
    .AddKafka(kafkaConfig, tags: new[] { "ready" });

For services depending on the messaging infrastructure covered in this series' RabbitMQ, Kafka, and Azure Service Bus guides, a health check confirming the broker connection is genuinely established — not just that the connection string is configured — catches a real class of "the app started but can't actually process its queue" failures before they manifest as a growing, unprocessed backlog.

Downstream service (HTTP/gRPC) health checks

builder.Services.AddHealthChecks()
    .AddUrlGroup(new Uri("https://inventory-service/health/ready"), name: "inventory-service", tags: new[] { "ready" });

Checking a downstream service's own health endpoint as part of this service's readiness check is worth doing deliberately and sparingly — it's genuinely useful for a hard dependency this service literally cannot function without, but chaining readiness checks too deeply across many services (Section 11) creates fragile, cascading failure coupling that undermines the very decoupling this series' Event-Driven Architecture and REST guides have advocated for elsewhere.

Weighing which dependencies deserve a readiness check

The right question for each dependency: "if this specific dependency is down, can this instance still successfully handle any meaningful fraction of its traffic?" If a dependency is used by only one rarely-hit endpoint, failing readiness for the entire instance because of it is disproportionate — a more nuanced approach (returning a degraded status just for that specific endpoint, or accepting that specific endpoint will simply error while the rest of the service continues serving traffic normally) is often the better design.

5. Custom Health Checks

Implementing IHealthCheck

public class QueueBacklogHealthCheck : IHealthCheck
{
    private readonly IQueueMetrics _queueMetrics;
    public QueueBacklogHealthCheck(IQueueMetrics queueMetrics) => _queueMetrics = queueMetrics;

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken)
    {
        var backlogSize = await _queueMetrics.GetBacklogSizeAsync(cancellationToken);

        if (backlogSize > 10000)
            return HealthCheckResult.Unhealthy($"Queue backlog critically high: {backlogSize}");
        if (backlogSize > 1000)
            return HealthCheckResult.Degraded($"Queue backlog elevated: {backlogSize}");

        return HealthCheckResult.Healthy();
    }
}

builder.Services.AddHealthChecks()
    .AddCheck<QueueBacklogHealthCheck>("queue-backlog", tags: new[] { "ready" });

Custom health checks let genuinely business-meaningful signals — not just generic connectivity — drive infrastructure decisions: this example connects directly to the queue-processing worker patterns covered in this series' Background Services guide, treating a dangerously large processing backlog as a degraded (or unhealthy) condition, potentially triggering an autoscale-out response (per this series' Cloud Cost Optimization guide) or alerting (per this series' Prometheus/Grafana guide) well before the backlog becomes an outright outage.

The three-state model: Healthy, Degraded, Unhealthy

return HealthCheckResult.Degraded("Non-critical cache is unavailable; falling back to database reads");

Degraded is a genuinely useful middle state, distinct from a binary healthy/unhealthy — it signals "this instance is working, but not optimally" without necessarily triggering the same drastic response (removal from load balancer rotation) that a full Unhealthy result would. How an orchestrator interprets Degraded varies (Kubernetes' binary probe model, Section 6, doesn't natively distinguish it the way a richer monitoring dashboard might), but it's valuable at minimum for the human-readable diagnostic response and for feeding into alerting/dashboarding systems that do distinguish it.

Health check UI for local development and debugging

builder.Services.AddHealthChecksUI().AddInMemoryStorage();
app.MapHealthChecksUI();

The AspNetCore.HealthChecks.UI package provides a simple dashboard visualizing the current status of every registered check over time — genuinely useful for local development and smaller deployments, though production environments typically rely on the dashboarding and alerting stack covered in this series' Prometheus/Grafana guide instead, since it integrates with the broader observability picture rather than being a separate, single-purpose tool.

6. How Kubernetes Consumes Health Checks

The three probe types, mapped directly to Section 2's three questions

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 15
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

startupProbe:
  httpGet:
    path: /health/live
    port: 8080
  failureThreshold: 30
  periodSeconds: 5

As covered in this series' Kubernetes/Helm guide, these three probe types map directly onto liveness, readiness, and startup from Section 2 — worth restating here specifically why the mapping matters: a failed liveness probe restarts the container; a failed readiness probe removes the pod from Service endpoints (stops routing traffic to it) without restarting; a startup probe delays both of the others from being evaluated until it succeeds, specifically accommodating slow-starting applications without weakening the liveness probe's steady-state sensitivity.

Why the startup probe exists as a distinct concept

# Without a startup probe, a slow-starting app needs an equally generous liveness initialDelaySeconds,
# which then makes liveness slow to detect a GENUINE hang once the app is past startup
startupProbe:
  failureThreshold: 30
  periodSeconds: 5   # allows up to 150 seconds for startup
livenessProbe:
  periodSeconds: 15
  failureThreshold: 3  # detects a genuine hang within 45 seconds, once past startup

This is the specific problem a startup probe solves: without it, accommodating a slow startup means either a very generous initialDelaySeconds on the liveness probe (which then also means liveness is slow to catch a genuine post-startup hang) or a liveness probe that's too aggressive during the legitimately slow startup window (causing restart loops on perfectly healthy, still-initializing instances) — the startup probe cleanly separates these two concerns.

Rolling updates depend entirely on readiness being correct

As covered in this series' Kubernetes/Helm guide, a rolling update relies on the readiness probe to determine when a newly-deployed pod is actually ready to receive traffic before terminating an old one — a readiness probe that returns healthy prematurely (before the application has genuinely finished initializing) is a direct, common cause of brief error spikes during otherwise-routine deployments, exactly the failure mode flagged in that guide's discussion of readiness probes and clean rollouts.

7. How Load Balancers and Cloud Platforms Consume Health Checks

Azure App Service and Application Gateway

Health check path configured on the App Service/Application Gateway → periodically polled →
  instances failing the check are automatically removed from the routing pool

As covered in this series' Azure Compute guide, App Service and Application Gateway support configuring a health check path — conceptually identical to Kubernetes' readiness probe, just implemented at the cloud platform's load-balancing layer rather than the orchestrator layer, and equally dependent on that endpoint reflecting genuine readiness rather than just process liveness.

AWS ELB/ALB and ECS

"healthCheck": {
  "command": ["CMD-SHELL", "curl -f http://localhost:8080/health/ready || exit 1"],
  "interval": 30,
  "timeout": 5,
  "retries": 3
}

As covered in this series' AWS Compute guide, an Application Load Balancer's target group health check determines which ECS tasks receive traffic, and ECS's own container-level health check (configurable in the task definition, as shown above) can additionally determine whether a task should be replaced entirely — the same liveness/readiness distinction from Section 2, expressed through AWS's specific mechanisms.

Docker's HEALTHCHECK instruction

HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:8080/health/live || exit 1

As covered in this series' Docker guide, this is a container-level (not orchestrator-level) health signal — docker ps reflects it directly, and Docker Compose's condition: service_healthy (also covered in that guide) depends on it for coordinating multi-container startup order, distinct from but complementary to whatever orchestrator-level probes (Kubernetes, ECS) might also be configured on top of the same underlying application.

8. Health Checks for Background Services

The gap: BackgroundService has no HTTP endpoint of its own

As covered in this series' Background Services guide, a BackgroundService-based worker often runs with no web server at all (a Worker Service project, per that guide) — meaning there's no natural place to expose /health/ready the way an ASP.NET Core web application has one by default.

Exposing health from a Worker Service

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddHealthChecks().AddCheck<QueueProcessorHealthCheck>("queue-processor");

var app = builder.Build(); // for a pure Worker Service, adding a minimal Kestrel listener specifically for /health is common

A common, pragmatic pattern: run a minimal HTTP listener within an otherwise non-web Worker Service specifically to expose a health endpoint — not to serve real application traffic, just to give Kubernetes (or whatever orchestrator manages the worker) something to probe, following the same probe-based lifecycle management the rest of this guide covers.

Health checks that reflect genuine progress, not just process liveness

public class QueueProcessorHealthCheck : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken)
    {
        var timeSinceLastSuccessfulPoll = DateTimeOffset.UtcNow - _lastSuccessfulPollTimestamp;
        return Task.FromResult(timeSinceLastSuccessfulPoll < TimeSpan.FromMinutes(5)
            ? HealthCheckResult.Healthy()
            : HealthCheckResult.Unhealthy($"No successful queue poll in {timeSinceLastSuccessfulPoll}"));
    }
}

This is precisely the pattern flagged (without a concrete implementation) in this series' Background Services guide's discussion of catching a "silently stuck" worker — the worker itself updates a shared "last successful iteration" timestamp on every successful loop cycle, and the health check simply confirms that timestamp is recent, catching a worker that's technically still running (the process hasn't crashed) but has stopped actually making progress, a failure mode a simple "is the process alive" check would never detect.

9. Health Checks vs. Deep Monitoring

What a health check is deliberately NOT

A health check answers a narrow, binary-ish (healthy/degraded/unhealthy) question, evaluated frequently (every 10-30 seconds is typical) and cheaply — it is deliberately not a substitute for the deeper observability covered in this series' OpenTelemetry, Prometheus/Grafana, and Distributed Tracing guides, which answer richer questions (why is latency elevated, what's the actual root cause, how has this trended over the past week) that a simple pass/fail check was never designed to answer.

Where the two overlap and complement each other

Health check fails → orchestrator takes an automated, immediate action (restart / remove from rotation)
Metrics/alerting fires → a human is notified to investigate, using traces/logs for root cause analysis

A well-designed system uses both together: health checks handle the fast, automated, "should this specific instance keep receiving traffic right now" decision with no human involved, while the broader observability stack handles the slower, richer "why is this happening, and what's the actual fix" investigation — conflating the two (trying to make a health check endpoint answer both questions) tends to produce either an overly expensive, slow health check (Section 10) or an under-informative monitoring dashboard.

Keeping health check logic and business logic separate

// ❌ Reusing complex business logic directly inside a health check is a common source of
// slow, fragile, or side-effect-carrying checks
public Task<HealthCheckResult> CheckHealthAsync(...) => _orderService.RunFullReconciliationAsync(); // way too heavy

// ✅ A health check should be lightweight and side-effect-free
public Task<HealthCheckResult> CheckHealthAsync(...) => _dbContext.Database.CanConnectAsync()
    ? Task.FromResult(HealthCheckResult.Healthy())
    : Task.FromResult(HealthCheckResult.Unhealthy());

A health check should be cheap and safe to run frequently, with no meaningful side effects — it's evaluated on a tight loop by infrastructure, potentially by multiple independent probes (Kubernetes' liveness, readiness, and a load balancer's own check, all simultaneously) simultaneously, so anything expensive or side-effect-carrying inside it compounds quickly.

10. Designing Health Checks That Don't Cause Outages

The cascading failure trap

Readiness check includes a full database query →
  database is genuinely struggling under load →
  EVERY instance's readiness check now also struggles/times out →
  load balancer removes EVERY instance from rotation simultaneously →
  the application is now FULLY down, when a slow-but-functioning database might have
  allowed at least some requests to succeed if traffic had kept flowing

This is a genuinely important, somewhat counterintuitive risk: an overly strict or overly expensive readiness check can turn a partial, gracefully-degrading problem into a complete outage, precisely because every instance's health check fails simultaneously and traffic stops entirely — worth weighing deliberately whether a dependency issue should actually take an instance fully out of rotation, versus letting it continue serving the requests it still can while some fraction inevitably error.

Timeouts on health check dependencies

builder.Services.AddHealthChecks()
    .AddSqlServer(connectionString, timeout: TimeSpan.FromSeconds(3), tags: new[] { "ready" });

A health check's own dependency calls need explicit, tight timeouts — a health check that hangs waiting on a slow database query doesn't just fail slowly, it can itself become a resource drain (accumulating hung requests) precisely during the kind of degraded conditions it exists to detect.

Avoiding a thundering herd on recovery

readinessProbe:
  periodSeconds: 10
  # Kubernetes doesn't natively stagger simultaneous probe timing across replicas —
  # worth being aware that many instances recovering in the same window can create
  # a simultaneous reconnection/traffic surge

When a shared dependency recovers after an outage, many instances' readiness checks can pass in roughly the same window, all resuming traffic (and reconnecting to the recovered dependency) nearly simultaneously — for genuinely sensitive downstream systems, this is worth considering alongside the broader resilience patterns (circuit breakers, gradual traffic ramp-up) that sit outside health checks themselves but interact directly with how quickly instances resume full traffic after a health check recovers.

Health check response time budget

A readiness check that itself takes several seconds to respond (because it's checking many dependencies sequentially) delays how quickly an orchestrator can make traffic-routing decisions — running dependency checks in parallel, and keeping the overall check's total time budget deliberately tight, keeps the health check itself from becoming a source of latency or a bottleneck under the exact load conditions it's meant to help manage.

11. Common Pitfalls

Quick Reference Table

Conclusion

Health checks look deceptively simple — an endpoint, a boolean-ish result — but the specific design decisions behind them (what liveness actually checks versus readiness, how tightly dependency checks are scoped and time-bounded, whether a partial issue should take an entire instance out of rotation) are what separate a health check that helps infrastructure make genuinely good decisions from one that actively causes or worsens outages. The single most important discipline, echoed throughout this guide's connections to the Kubernetes/Helm, Docker, and cloud compute guides elsewhere in this series, is keeping liveness and readiness conceptually and practically distinct: liveness for "is this fundamentally broken enough to restart," readiness for "can this handle traffic right now," and never conflating the two into one endpoint that ends up answering neither question well.

Done right, health checks are the quiet, automated foundation that makes rolling deployments, autoscaling, and self-healing actually work reliably — connecting directly to nearly every other operational guide in this series, from CI/CD deployment strategies to Background Services' worker monitoring to the observability stack that takes over once a health check signals something is genuinely wrong and a human needs to investigate why.

Found this useful? Feel free to star the repo, open an issue with corrections, or share the restart storm that taught you to separate liveness from readiness for good.