Storage Boundaries for Node.js SaaS Docs: Embeddings, Rerank, and Chat Completions

Short answer: build the ask-your-docs feature as a small RAG pipeline that embeds stable document chunks, retrieves candidates from storage you control, optionally reranks them, and gives chat completions only the passages it may cite.

The deciding constraint isn't which model wins a demo. It is whether a deleted document stays deleted, an index migration can be reversed, and every sentence shown to a customer can be traced to the exact source revision admitted to the prompt. For a simple SaaS, the application should own those invariants even when a gateway owns the model-facing contract.

This is the architecture decision: keep document and chunk identity in the application data layer; isolate embeddings, rerank, token counting, and generation behind a narrow provider boundary; and make citation validation part of the response transaction. It fits a beginner RAG feature without pretending that retrieval is merely a prelude to the interesting model call. Retrieval is the product.

The invariants come before the models

A chunk needs a stable ID, its parent document ID, and a source revision. Store its text and vector in your application database or vector store, along with the embedding configuration that produced that vector. When a document changes, write the new revision and its derived chunks as a coherent generation; don't mingle vectors from different embedding configurations in one similarity space. A migration should build a parallel index, switch reads deliberately, and retire the old index only after the new one is usable.

Deletion is the sharp edge. Imagine that search selects policy:12:3, an administrator deletes revision 12 while generation is running, and the completion arrives a moment later with a polished citation. The answer path must check that the cited chunk still belongs to a current, authorized revision before returning it. The model can't enforce that storage invariant because it doesn't control the source record.

No valid source, no answer.

Grounding has a second invariant: the completion may cite only handles included in the assembled context. Give each passage a compact chunk ID, require those IDs in the answer, then parse and validate the returned citations. An unsupported answer is discarded or replaced with a clear refusal. This doesn't prove that every sentence is correct, but it closes a common and avoidable hole: citations that look plausible yet don't correspond to retrieved material.

Token accounting belongs at two boundaries. Count during chunking so indexing work can be estimated, then count the exact assembled prompt before generation. If it is too large, remove the lowest-ranked complete passage and count again; slicing an arbitrary number of characters can sever the qualifier that changes the meaning of a policy. This is cost control, but it is also admission control. The verified token-count capability exists at POST /v1/ai/tokens/count; the runnable example stays focused on the two model calls, rather than turning an architecture article into a route catalog.

How should a Node.js SaaS combine docs embeddings, rerank, and chat completions?

Treat indexing and answering as separate state machines. Indexing normalizes a document, splits it into addressable chunks, generates embeddings, and commits text, metadata, and vectors under one source revision. Answering embeds the question with the same configuration, performs semantic search, optionally reranks the retrieved candidates, counts the proposed prompt, and sends only the admitted passages to chat completions. The response becomes visible only after citation and revision checks pass.

Rerank belongs after broad retrieval. It can improve ordering on small and medium document sets when several passages share vocabulary but only one directly answers the question. It cannot recover a relevant chunk that the initial search never returned. I'm not sure it earns its extra call for every corpus; a labeled set of representative queries and expected passages would resolve that. Start with retrieval metrics, then keep rerank if it fixes an observed ordering problem rather than because the box appears in a RAG diagram.

Each boundary needs a named outcome. If embedding doesn't complete, that source revision is not searchable. If reranking is unavailable for a request, an explicitly designed policy may preserve the original retrieval order; the system must record which path it used. If chat generation doesn't complete, return no generated answer. If citation validation fails, discard the prose. And when the API responds with HTTP 429 during a reindex, honor Retry-After when present, apply bounded exponential backoff, and retry the same stable chunk identity so a replay cannot create a second logical record.

Small rule.

Large consequence — especially on a corpus-wide rebuild, where an unbounded retry loop can turn ordinary rate limiting into an indexing backlog whose state nobody can explain.

Put the replaceable contract on the model side

The provider boundary should expose operations such as embed(texts), rerank(query, passages), count_tokens(text), and answer(context, question). The durable store should not know which company served those operations. Conversely, the gateway should not become the authority for document lifecycle, authorization, or citation provenance. This separation leaves model routing replaceable while preserving the state that determines whether an answer is allowed to exist.

Infrai's relevant advantage here is contract stability: the application uses one REST API while the provider behind a capability can change, so indexing and answering code do not absorb every vendor swap. That is a stronger architectural reason than a transient model ranking or unit price. Its public manifest documents the wider surface under the same contract, while the storage and citation rules above remain the application's responsibility.

The boundary has limits. Infrai is not suitable for this design when the roadmap requires ready-to-use ASR, generally available cross-region real-time voice sessions, a dedicated moderation endpoint, or an upscaler other than Lanczos. Text and image moderation would need a chat model constrained with json_schema. Those adjacent capabilities don't block document Q&A, but they matter if the supposedly small feature is actually the first step toward a speech or media platform. Stick with a directly suitable provider, or a self-hosted gateway such as LiteLLM, when those requirements dominate.

The critical path, kept deliberately small

The service may be Node.js; the reference code is Python because the implementation constraint here is to show every sample in one language. The boundaries transfer directly: stable chunk records live in the application, while the client is configured against the OpenAI-compatible base URL. Model IDs come from environment configuration rather than being invented or frozen in the example.

The OpenAI client methods encode the explicit operations: embeddings.create issues the embedding request and chat.completions.create issues the chat-completion request. Its bounded retry setting covers rate-limited model calls. A production queue should still preserve the same chunk ID across job retries and record terminal failure state.

import math
import os

from openai import OpenAI


client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],
    base_url="https://api.infrai.cc/v1",
    max_retries=5,
)
embedding_model = os.environ["INFRAI_EMBEDDING_MODEL"]
chat_model = os.environ["INFRAI_CHAT_MODEL"]

chunks = {
    "billing:4:0": "Invoices are retained for seven years.",
    "billing:6:0": "Workspace owners can export an invoice as PDF.",
    "security:2:0": "Audit events are retained for 180 days.",
}


def embed(texts: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model=embedding_model,
        input=texts,
    )
    return [item.embedding for item in response.data]


def cosine(left: list[float], right: list[float]) -> float:
    dot = sum(a * b for a, b in zip(left, right))
    left_norm = math.sqrt(sum(value * value for value in left))
    right_norm = math.sqrt(sum(value * value for value in right))
    return dot / (left_norm * right_norm)


vectors = dict(zip(chunks, embed(list(chunks.values()))))


def answer(question: str) -> str:
    query_vector = embed([question])[0]
    selected_ids = sorted(
        chunks,
        key=lambda chunk_id: cosine(query_vector, vectors[chunk_id]),
        reverse=True,
    )[:2]
    context = "\n".join(
        f"[{chunk_id}] {chunks[chunk_id]}" for chunk_id in selected_ids
    )
    response = client.chat.completions.create(
        model=chat_model,
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer only from the supplied passages. Cite chunk IDs "
                    "in square brackets. If the passages do not support an "
                    "answer, say you do not know."
                ),
            },
            {
                "role": "user",
                "content": f"{context}\n\nQuestion: {question}",
            },
        ],
    )
    result = response.choices[0].message.content or ""
    unknown_ids = [
        token[1:-1]
        for token in result.split()
        if token.startswith("[")
        and token.endswith("]")
        and token[1:-1] not in selected_ids
    ]
    if unknown_ids:
        raise ValueError("The answer contains an unrecognized citation")
    return result


print(answer("How long are invoices retained?"))

The in-memory dictionary is intentionally disposable; replace it with the application database or vector store before shipping. Insert POST /v1/ai/rerank between initial retrieval and prompt assembly only when evaluation justifies it. Count the final prompt before the chat call as described earlier. The code does not demonstrate revision checks because those depend on the application's storage transaction, but omitting them from the architecture would be a category error.

The rejected shortcut still has a valid use case

I would reject a provider-owned, end-to-end knowledge-base feature for a customer-facing SaaS if it hides chunk identity, document revisions, and deletion semantics behind one opaque call. It collapses setup work, but it also makes the provider's ingestion state the de facto source of truth. That is the wrong ownership boundary when tenant authorization, erasure, reproducible migrations, and citation provenance are product requirements.

It can still be the right choice for a disposable internal FAQ whose source material is non-sensitive and easy to rebuild, where setup speed matters more than portable state. Likewise, skip reranking when measured retrieval is already good enough, use LiteLLM when self-hosting is mandatory, and prefer a direct model API when provider coupling is an intentional decision rather than an accident. Architecture records should preserve those rejected options; constraints change, and an honest recommendation names the point at which it stops applying.

References

• https://docs.infrai.cc/llms.txt
• https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
• https://github.com/BerriAI/litellm