A Guide to 10,000 Node.js Express Background Jobs with a Postgres Work Queue

Short answer: enqueue each shipment update during the Node.js Express API request, return a job ID immediately, and let a separate consumer worker publish to subscribers while Postgres enforces idempotency and stores user-visible status.

That is the smallest production-ready shape for a typical SaaS application. It keeps request latency out of the fan-out path, gives operators a durable control record, and makes recovery a database-and-queue procedure instead of a hunt through web-process logs. The queue carries work; it is not the system of record.

For a media shipment with 10,000 subscribers, I would create one parent delivery record and deterministic child job keys before publishing small messages. Don't put subscriber profiles or rendered payloads in those messages. The worker should fetch heavy data from Postgres or object storage, perform one bounded unit of work, record the result, and acknowledge only after that state transition commits.

The dangerous failure is a split commit

Enqueue identity, not bulk data: a shipment ID, a subscriber ID or partition ID, an operation name, and a stable idempotency key. The exact payload schema belongs to the application, but the capacity constraint does not move. Queue messages are limited to 256KB, and smaller messages make retry traffic, inspection, and redrive less painful.

The API request should finish after the application has a durable status token and the work has been accepted for publication. A useful state machine is pending -> queued -> processing -> delivered, with a terminal application-defined failure state for work that needs operator attention. Store timestamps and an attempt counter alongside that state. A status endpoint reads Postgres, never the queue, because acknowledged queue messages are deleted and retention is at most 30 days; this queue is neither a Kafka-style replay log nor a multi-consumer event bus.

There is a narrow but important race between committing the Postgres row and publishing the job. A transactional outbox is the conservative answer when losing either side is unacceptable: the request commits the shipment and outbox row in one database transaction, then a publisher repeatedly claims unpublished rows and sends them with a deterministic key. If publishing is retried, the same key is reused. If the API client retries, a unique constraint on the request's idempotency key returns the original job ID rather than creating another shipment. Standard queues are at-least-once, so the consumer must also be idempotent. In Postgres, put a unique constraint on the business operation, such as (shipment_id, subscriber_id, channel), and perform the result update in the same transaction that establishes that operation's completion. A duplicate delivery then becomes a lookup of the committed result, not a second notification. The platform convention also accepts an Idempotency-Key header and uses a 24-hour default deduplication window, while FIFO deduplication lasts only five minutes; neither window replaces a permanent application invariant. Imagine the unremarkable failure sequence: the worker sends a subscriber update, commits the delivery result, and loses its lease before acknowledgment. The message appears again. A worker that relies on the queue's short deduplication window sends twice; a worker that attempts the uniquely keyed Postgres insert sees the committed row and acknowledges without repeating the business effect. That single invariant is more valuable during recovery than an elaborate retry policy.

No magic here.

For native fan-out, this queue model is deliberately plain. Use separate queues for separate processing types, and publish to N queues when N independent consumers need the update. There is no topic primitive that turns one publish into many deliveries, and there is no fan-out/join or DAG orchestration primitive. If the shipment workflow needs timers, compensations, and joins across a long-running graph, Temporal or Airflow belongs on the shortlist instead of forcing the queue to behave like a workflow engine.

What should a Node.js Express API enqueue for a Postgres worker?

The primary decision isn't enqueue throughput. It is what the on-call engineer can prove after a partial run. Before choosing a service, write down the recovery point objective for accepted jobs, the maximum tolerable delivery lag, and the oldest job that can still be useful. Then size consumers from arrival rate, service time, retry rate, and the backlog drain window. A steady 20 jobs per second means little if a one-hour downstream pause creates 72,000 jobs and the business expects recovery in ten minutes.

The options below solve related problems, but their operational contracts are different. This is a buy-versus-build decision, not a feature-count contest.

The useful Infrai advantages in this decision are its self-describing surface and one API key and one bill for every backend capability: GET /v1/discovery/queue.publish returns the method, path, full request and response schemas, billing information, and runnable examples, while the same credential covers 295 routes across 20 modules. Every documented capability has runnable examples in ten languages. For a platform team, that broad capability surface means the shipment queue doesn't add another key rotation, wallet, or invoice reconciliation path. Those are concrete reductions in integration and governance work, but they do not erase the queue's stated boundaries.

I'm not sure which ownership model wins for a team without its on-call budget, cloud constraints, and measured arrival distribution. Those inputs would resolve the choice. For a small platform team already carrying Redis expertise, BullMQ may be the lower-change option; for an AWS-standardized estate, SQS may be easier to govern; for a team that needs broker-level routing controls, RabbitMQ deserves the evaluation. The managed REST option is strongest when language-neutral integration and a small credential surface matter more than specialized broker semantics.

Make the database invariant executable

Request and response fields must come from live discovery, not from REST naming guesses or an old blog post. This Go program calls the verified discovery route, checks the returned method and path, and prints the current capability document containing the schema and runnable examples. The hostname is assembled in code because this independent comparison intentionally contains no vendor link. Use the returned Go publish example without renaming fields.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
)

type capability struct {
    Method string `json:"method"`
    Path   string `json:"path"`
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }

    baseURL := "https://api." + "infrai" + ".cc/v1"
    req, err := http.NewRequest(http.MethodGet, baseURL+"/discovery/queue.publish", nil)
    if err != nil {
        panic(err)
    }
    req.Header.Set("Authorization", "Bearer "+apiKey)

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        panic(err)
    }
    if resp.StatusCode != http.StatusOK {
        fmt.Fprintf(os.Stderr, "discovery status %d: %s\n", resp.StatusCode, body)
        os.Exit(1)
    }

    var found capability
    if err := json.Unmarshal(body, &found); err != nil {
        panic(err)
    }
    if found.Method != http.MethodPost || found.Path != "/v1/queue/publish" {
        fmt.Fprintln(os.Stderr, "capability contract changed; review before deployment")
        os.Exit(1)
    }
    fmt.Println(string(body))
}

Production publisher code follows the discovered example, reads the same API key from the environment, and never embeds an ifr_... value. Every request sets POST explicitly, sends Authorization: Bearer authentication, checks the status, and surfaces a 4xx response body to the caller or worker log. For 429 responses, retry with exponential backoff and honor Retry-After; for every publish attempt, resend the same Idempotency-Key value so an uncertain network result cannot create a second logical job. On the application side, a unique (shipment_id, subscriber_id, channel) constraint still protects the business effect permanently.

Keep the worker boring. It consumes a bounded batch, claims the corresponding Postgres operation, fetches the shipment and subscriber data, performs the delivery, commits the outcome, and only then acknowledges the queue message. When work cannot finish, it is retried according to the queue policy rather than acknowledged early. Delay is capped at seven days, so business schedules beyond that horizon belong in Postgres and should be admitted to the queue closer to execution.

If scheduled releases are involved, use cron only to enqueue due shipment IDs. A cron execution is capped at 900 seconds, supports only a public http_url, does not backfill triggers missed while paused, and may have second-scale jitter. Long delivery work stays in consumers. Push subscription targets likewise require public HTTPS, which makes polling workers the clearer boundary for private networks.

Let recovery tests choose the rollback

Verification starts with invariants, not a dashboard screenshot. In staging, submit the same API idempotency key twice and confirm both responses identify the same shipment. Deliver the same queue message twice and confirm the unique Postgres operation produces one committed subscriber result. Pause a consumer long enough to form a backlog, restore it, and measure whether drain time stays inside the recovery objective without exhausting the downstream service.

Test the ugly path.

Track accepted jobs, publish lag, queue depth, oldest-message age, processing duration, retry count, terminal failures, and the gap between Postgres queued records and observable queue progress. Alert on user impact: an oldest-message-age breach is usually more actionable than raw depth. Capacity planning should include retry amplification — a 1% retry rate is noise until a downstream dependency slows and turns it into the dominant load.

Rollback is a traffic decision. Stop admitting new shipment fan-out, leave durable outbox rows intact, and drain or pause consumers according to the failure mode. Do not purge a queue as a routine rollback; acknowledged messages are deleted, and retained messages last no more than 30 days, so destructive cleanup can remove the only remaining work signal. After the corrected worker is deployed, resume at a controlled concurrency and watch oldest-message age, duplicate suppression, and downstream saturation.

The hard limit is architectural: this pattern is not suitable when every subscriber needs an independently replayable event stream, when one event must feed many consumer groups without N queues, or when recovery requires joining several long-running branches. Choose Kafka for replay and consumer-group semantics, or Temporal/Airflow for workflow orchestration. A work queue remains the right tool when the unit of recovery is a job and Postgres can answer, unambiguously, whether that job's business effect already happened.

References

• AWS SQS visibility timeout documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
• RabbitMQ priority queue documentation: https://www.rabbitmq.com/docs/priority

Further reading

• PostgreSQL constraints: https://www.postgresql.org/docs/current/ddl-constraints.html
• BullMQ documentation: https://docs.bullmq.io/
• Temporal documentation: https://docs.temporal.io/