Cheap Node.js LLM Moderation: Estimate Token Cost Before User Content Checks

Short answer: for cheap LLM moderation in Node.js, estimate token cost before classifying user content, reserve against an explicit input-and-output ceiling, and commit one schema-validated verdict per revision. Treat the estimate as a scheduling guardrail, not a promised invoice.

I learned the shape of this problem from a production page. A worker lost its acknowledgement after a timeout, then retried; the classifier ran twice and the downstream notifier saw two state transitions. The log showed an HTTP 409 from the conditional write, which was the useful part of the incident: the second computation was harmless, but the second commit was not. The runbook now treats retries as normal and side effects as precious.

The invariant is simple: the same content digest, policy revision, and model revision may produce many attempts, but only one durable moderation decision. Everything below follows from enforcing that invariant before a queue can amplify the mistake.

What should a Node.js moderation gate measure before classifying text and images?

The ingress service should measure three different things before it enqueues work: payload shape, model accounting units, and the allowed output envelope. Payload shape covers byte limits, image count, dimensions, and media type. Accounting units are model-specific text tokens plus whatever units the selected model uses for an image. Output is a ceiling for the typed verdict, not an invitation to generate an essay.

Do not estimate text with length / 4 and call it done. Tokenizers differ, and image accounting can depend on preprocessing or detail settings. A counter owned by the model adapter is a better boundary: Node.js validates the request, then asks the adapter for counts and a versioned rate record. If the adapter cannot identify its tokenizer or image policy, the request should wait for configuration rather than enter the paid lane on a guess.

The estimate is an admission decision. In integer micro-units, a conservative reservation is:

ceil(input_units * input_rate / 1,000,000) + ceil(max_output_units * output_rate / 1,000,000)

Store the inputs to that calculation with the job. Later, reconcile the reservation with usage reported by the classifier. Actual output may stop early; a rate table may change; and two image transforms with the same file size can have different accounting. I'm not sure one global safety margin fits every tenant, so I would measure estimate error by model and payload class before choosing one. Your mileage may vary.

The failure path is a state machine, not a single API call

Admission, classification, validation, commit, and publication are separate states. A queue lease can expire while classification is still running. A client can retry after a 504 even though the first attempt finished. A deploy can replay an old message under a new policy. Each transition needs a durable key and an observable deadline.

Retries happen.

I use a key derived from the normalized content digest, policy revision, and model revision. A policy edit is therefore a new decision by design. The consumer inserts the verdict with a uniqueness constraint on that key, and only the transaction that wins may create an outbox event. Acknowledgement follows the commit. Never acknowledge between schema validation and the conditional insert.

The compact result should be boring: allow, review, or block; a bounded array of reason codes; and the policy revision. JSON Schema makes that contract inspectable, while server-side validation remains mandatory because a schema in a prompt is not enforcement. Unknown fields should be rejected or ignored deliberately, not propagated accidentally into policy code.

A Go preflight keeps accounting testable

The web tier can remain Node.js while a small worker owns counting and commit semantics. The counter is injected so tests can use a fixed tokenizer fixture and production can select the correct model adapter. This is deliberately a narrow example; the storage transaction belongs beside the idempotency constraint.

package moderation

import (
    "context"
    "errors"
)

type Input struct {
    Text      string
    ImageRefs []string
}

type Counter interface {
    TextUnits(context.Context, string) (int64, error)
    ImageUnits(context.Context, []string) (int64, error)
}

type Rates struct {
    InputMicrosPerMillion  int64
    OutputMicrosPerMillion int64
}

type Estimate struct {
    InputUnits     int64 `json:"input_units"`
    MaxOutputUnits int64 `json:"max_output_units"`
    ReservedMicros int64 `json:"reserved_micros"`
}

func Preflight(ctx context.Context, in Input, counter Counter, rates Rates, maxOutput int64) (Estimate, error) {
    if counter == nil || maxOutput <= 0 || rates.InputMicrosPerMillion < 0 || rates.OutputMicrosPerMillion < 0 {
        return Estimate{}, errors.New("invalid moderation budget configuration")
    }
    textUnits, err := counter.TextUnits(ctx, in.Text)
    if err != nil {
        return Estimate{}, err
    }
    imageUnits, err := counter.ImageUnits(ctx, in.ImageRefs)
    if err != nil {
        return Estimate{}, err
    }
    if textUnits < 0 || imageUnits < 0 {
        return Estimate{}, errors.New("counter returned negative units")
    }
    input := textUnits + imageUnits
    reserved := ceilPerMillion(input, rates.InputMicrosPerMillion) +
        ceilPerMillion(maxOutput, rates.OutputMicrosPerMillion)
    return Estimate{InputUnits: input, MaxOutputUnits: maxOutput, ReservedMicros: reserved}, nil
}

func ceilPerMillion(units, microsPerMillion int64) int64 {
    if units == 0 || microsPerMillion == 0 {
        return 0
    }
    return (units*microsPerMillion + 999_999) / 1_000_000
}

type Verdict struct {
    Decision       string   `json:"decision"`
    Reasons        []string `json:"reasons"`
    PolicyRevision string   `json:"policy_revision"`
}

const VerdictSchema = `{
  "type": "object",
  "additionalProperties": false,
  "required": ["decision", "reasons", "policy_revision"],
  "properties": {
    "decision": {"type": "string", "enum": ["allow", "review", "block"]},
    "reasons": {"type": "array", "items": {"type": "string"}, "maxItems": 8},
    "policy_revision": {"type": "string", "minLength": 1}
  }
}`

In production, guard multiplication overflow and reject a counter result that exceeds the configured payload ceiling. Decode the response, validate the schema, write by idempotency key, and publish from an outbox. The order matters more than the particular database or queue.

Where this approach does not fit

An estimate is not suitable when a workflow requires an exact final charge before it starts; only post-classification usage can provide that, so use a reconciliation hold and accept the timing trade-off. Stick with a synchronous transaction when traffic is small, callers can wait, and every side effect already shares one commit boundary. A queue brings leases, replay, backlog alarms, and operational work.

Text-first routing can save image work only when policy explicitly allows text to settle a case. If every image must be examined, staging adds delay without removing the required call. Human review also deserves its own state and deadline; do not squeeze an appeal workflow into a three-value automated verdict.

The operational checklist is intentionally plain: replay a fixed corpus for each policy and model revision, inject duplicate delivery, expire leases, cancel in-flight work, and verify schema rejection. Canary by tenant while watching queue age, classification latency, review rate, conditional-write conflicts, and forecast error. Queue age is the page-worthy signal because it maps to a user deadline; raw depth alone does not.

No magic here.

The same accounting boundary should not be reused blindly for reranking or speech recognition. Reranking orders candidates, while speech recognition turns audio into text; each has different units, latency, and retry semantics. Keeping those budgets and revisions separate prevents an unrelated workload from quietly consuming the moderation lane.

References

• https://docs.cohere.com/docs/rerank-overview
• https://github.com/openai/whisper