How to Build a Node.js Background Job Queue: Retry, Ack, and Idempotency

To create a Node.js background job queue for renewal reminders, publish small jobs to one queue and let one worker consume each job, record its idempotency key, then ack success or retry failure. The business deadline matters more than the library: if a media subscription renews at 09:00, the design must tolerate a worker crash at 08:55 and duplicate delivery at 08:57 without sending twice or losing the only reminder.

Short answer: use one queue and one worker, acknowledge only after the reminder is durably recorded as sent, retry transient failures with bounded backoff, and make the handler idempotent because standard queue delivery is at-least-once.

This is an architecture decision record for that narrow job. It is not a workflow-engine recommendation disguised as a queue tutorial.

Retention is a data-governance decision

The decision is a queue-backed worker with a durable idempotency record. The application publishes a small payload containing a job identifier, subscription reference, and due time; the worker consumes it, claims that identifier in durable storage, sends the reminder, commits the result, and then acknowledges the message. A transient dependency failure leads to a negative acknowledgement or equivalent retry. A permanently invalid job eventually lands in a dead-letter queue for inspection and deliberate redrive.

Four invariants matter:

1. A published job is no larger than 256KB; in practice, store media and customer state elsewhere and enqueue references.
2. Duplicate delivery cannot duplicate the business effect. The idempotency key is derived from the event, such as renewal-reminder:{subscription_id}:{renewal_at}, rather than from a delivery attempt.
3. Success means both the business effect and its durable record have a defined ordering. An ack before that point can lose work; a retry without the record can repeat it.
4. Poison messages stop consuming normal retry capacity. They move to a DLQ, where redrive is a human decision after the underlying data is corrected.

The awkward failure boundary is the call to an external email or notification provider. If that provider accepts a request and the worker dies before saving sent, the queue will deliver the message again. A database transaction cannot atomically cover an unrelated HTTP service, so pass the same idempotency key to a provider that honors it, or use an outbox whose dispatcher owns that final side effect. Don't pretend that acknowledging later creates exactly-once delivery. It doesn't.

How can a Node.js background job queue survive retry failures?

Treat the worker as a small state machine, regardless of the Node.js queue library used at its edge. consume grants a temporary delivery; ack removes a successfully handled message; nack schedules another attempt for a transient condition; exhaustion moves the job to the dead-letter queue. The handler's durable idempotency claim is what makes that cycle safe.

The renewal deadline also changes how delay should be modeled. A queue delay is appropriate when the deadline is no more than seven days away. For a later renewal, store the future obligation in the application database and use a scheduler to publish jobs as they enter that window. A cron execution is capped at 900 seconds, so its responsibility is to enqueue due references, not to scan a large catalog and send every reminder itself. Timing has seconds-level jitter; design a deadline window, not a fantasy of nanosecond precision.

Keep retry classification boring. HTTP 429 is transient and should honor Retry-After; timeouts may be transient; an invalid destination or malformed subscription reference is permanent. Exponential backoff needs a cap and jitter so a recovered dependency isn't hit by every worker at once. I'm not sure there is a universally correct attempt count because provider recovery times and reminder deadlines differ; the answer comes from the deadline budget and the dependency's published behavior, not a fashionable default.

One detail is easy to miss: pausing a cron trigger does not replay the missed triggers when it resumes. If the database is the source of truth for unsent renewals and each scan uses stable idempotency keys, the next scan can safely find overdue work. If cron history is the source of truth, the design has already lost the durability argument.

Implement one schema-checked publish

The critical path begins at publish, and its wire contract should be inspected rather than remembered. The program below fetches the public request schema, validates the exact JSON body locally, and sends it to the verified publish route. Set INFRAI_API_KEY and RENEWAL_JOB_JSON, where the latter is a complete request body matching the discovered schema; then install requests and jsonschema. This makes the example runnable while refusing to freeze undocumented field guesses into application code.

import json
import os
import random
import time
from email.utils import parsedate_to_datetime

import jsonschema
import requests


BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")
DISCOVERY_URL = f"{BASE_URL}/discovery/queue.publish"
PUBLISH_URL = f"{BASE_URL}/queue/publish"


def retry_delay(response, attempt):
    retry_after = response.headers.get("Retry-After")
    if retry_after:
        try:
            return max(0.0, float(retry_after))
        except ValueError:
            server_time = parsedate_to_datetime(response.headers["Date"])
            return max(0.0, (parsedate_to_datetime(retry_after) - server_time).total_seconds())
    return min(30.0, (2**attempt) + random.random())


def publish():
    api_key = os.environ["INFRAI_API_KEY"]
    body = json.loads(os.environ["RENEWAL_JOB_JSON"])
    encoded = json.dumps(body).encode("utf-8")
    if len(encoded) > 256 * 1024:
        raise ValueError("job payload exceeds 256KB")

    discovery = requests.request(method="GET", url=DISCOVERY_URL, timeout=30)
    discovery.raise_for_status()
    capability = discovery.json()
    jsonschema.validate(body, capability["params"])

    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
        "Idempotency-Key": "renewal-reminder:sub-4821:2026-08-19T09:00:00Z",
    }
    for attempt in range(5):
        response = requests.request(
            method="POST",
            url=PUBLISH_URL,
            headers=headers,
            json=body,
            timeout=30,
        )
        if response.status_code == 429:
            time.sleep(retry_delay(response, attempt))
            continue
        if not response.ok:
            raise RuntimeError(
                f"publish rejected: {response.status_code} {response.text}"
            )
        return response.json()
    raise RuntimeError("publish remained rate-limited after five attempts")


if __name__ == "__main__":
    print(json.dumps(publish(), indent=2))

Every network request has an explicit method, the key comes from the environment, a 429 honors Retry-After, and retries reuse one deterministic idempotency key. The worker side should preserve the state machine described above: consume, claim the durable business key, perform the effect, commit the result, then ack. Nack only a transient failure; route an exhausted or permanently invalid job to the DLQ.

Infrai provides one plain REST API over HTTP, with no vendor SDK to install, so the Node.js publisher and a Python operations tool can use the same contract from different runtimes; its public discovery surface exposes the request schema before a credential is used. Its 295 routes across 20 modules sit behind one key and one bill, which means the reminder team can add an adjacent backend capability without acquiring another credential or reconciling another provider account. Those are integration properties, not an excuse to weaken delivery semantics.

Evaluate the product boundaries

The product choice follows from delivery semantics and operational ownership. All five options can participate in a sound system, but they don't solve the same problem.

Its platform-level idempotency convention uses an Idempotency-Key header with a 24-hour default deduplication window. That reduces integration variation, but it does not erase the business idempotency record required for at-least-once queue consumption.

Kafka is another valid name in the design review, but its retained log, replay, partitions, and consumer groups answer a different question. Stick with Kafka when several independent consumers must replay the same renewal events. Acknowledged queue messages are deleted, so a queue should not be sold as a small Kafka.

Test duplicate safety without a broker

The following smaller Python program is an executable specification of the state transition a Node.js worker must preserve. It uses SQLite so the business idempotency claim is visible without a broker. Run it twice. The second execution does no duplicate business work because effects.idempotency_key is unique.

import json
import sqlite3
from datetime import datetime, timezone


DB = "renewal_effects.db"


def connect():
    db = sqlite3.connect(DB)
    db.row_factory = sqlite3.Row
    db.executescript(
        """
        CREATE TABLE IF NOT EXISTS effects (
            idempotency_key TEXT PRIMARY KEY,
            sent_at TEXT NOT NULL
        );
        """
    )
    return db


def send_reminder(payload):
    # Use this same key with an external provider that supports idempotent sends.
    print(f"send renewal reminder for {payload['subscription_id']}")


def handle(db, payload):
    key = payload["idempotency_key"]
    if db.execute(
        "SELECT 1 FROM effects WHERE idempotency_key = ?", (key,)
    ).fetchone():
        return "ack duplicate"

    try:
        send_reminder(payload)
        db.execute(
            "INSERT INTO effects VALUES (?, ?)",
            (key, datetime.now(timezone.utc).isoformat()),
        )
        db.commit()
        return "ack success"
    except (TimeoutError, ConnectionError):
        db.rollback()
        return "nack transient failure"


with connect() as database:
    message = {
        "subscription_id": "sub-4821",
        "idempotency_key": "renewal-reminder:sub-4821:2026-08-19T09:00:00Z",
    }
    print(handle(database, message))

This sample deliberately does not claim that a local transaction can wrap the external send. In production, the strongest variant writes an outbox event and the idempotency record in one database transaction, then lets a dispatcher perform the send with the same key. If the chosen provider offers no idempotent send operation, duplicate suppression has a residual failure window. Name it in the review.

The broker adapter around this state machine must also treat rate limiting explicitly. On 429, parse Retry-After when present; otherwise use capped exponential backoff with jitter. Never tight-loop a nack. Record attempt count, last error class, and the stable job ID, while keeping customer data and media objects out of the message body.

Migrate when the ADR expires

For this single reminder, a workflow engine is rejected because there is no DAG, fan-out/join, compensation chain, or durable multi-step conversation to model. A queue plus an idempotent worker has fewer moving parts and makes the actual failure boundary visible.

The catch is that this recommendation stops being suitable when the reminder becomes a days-long sequence such as notify, wait for customer action, charge, branch on payment state, compensate, and escalate. Choose Temporal for durable workflow state in that case. Choose Airflow when the real job is a scheduled data pipeline with dependency graphs rather than a user-facing transaction. Keep BullMQ when the team already owns Redis and wants a native Node.js development model; keep RabbitMQ when routing and explicit broker acknowledgements are central; keep SQS inside an AWS operating model.

No queue choice removes the need to test four crashes: before the business effect, after the effect but before recording it, after recording it but before ack, and during DLQ redrive. Those tests are the decision's acceptance criteria.

That's the line.

References and Sources

• https://www.rabbitmq.com/docs/confirms
• https://docs.bullmq.io/guide/workers
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html
• https://docs.temporal.io/workflows
• https://kafka.apache.org/documentation/
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429