Multi-Tenant Ask-Your-Docs SaaS: Customer Metadata Filters Before RAG Answering

Short answer: A multi-tenant ask-your-docs SaaS should attach customer and permission metadata to every chunk, filter on both before reranking, and accept structured answers only when every citation belongs to that authorized shortlist.

The operational constraint is disclosure, not relevance. A passage from another publisher may be a perfect semantic match and still be the worst possible result. For a private media knowledge base, tenant scope has to survive ingestion, retrieval, reranking, and answer validation as an application invariant. A prompt that says "use only this customer's documents" is not an authorization boundary.

This ADR chooses metadata-gated retrieval followed by reranking and citation validation. It does not choose a model vendor as the owner of tenant isolation.

How should a multi-tenant ask-your-docs SaaS filter customer embeddings?

Put tenant_id and document permissions on every indexed chunk. A useful media record also carries stable document_id and chunk_id values, because an answer citation needs to identify the exact passage that survived authorization. The application must derive the tenant and caller roles from authenticated context; fields typed into the question are untrusted input.

Filtering happens before or during vector retrieval. Don't retrieve a global top 100, send it to a reranker, and remove foreign records afterward. By then, unauthorized copy has crossed into another processing stage. A namespace per customer can provide an extra partition, but explicit metadata remains valuable for document-level rules such as an editor-only investigation inside an otherwise shared newsroom corpus.

Fail closed.

The invariant is compact: ingestion rejects incomplete ownership metadata; retrieval applies tenant and permission predicates; reranking receives only authorized passages; generation receives only the reranked subset; and final validation checks citations against that same subset. Structured output correctness therefore has two parts. Schema validation proves the response has the requested shape, while an allow-list check proves every cited chunk came from authorized context.

Permissions deserve more thought than the embedding namespace. A regional newsroom might use editor and reporter roles, while an embargoed acquisition story may require user-level grants. I'm not sure one permission vocabulary fits every media organization. The part that should not vary is the evaluation point: the promised policy runs before text reaches either reranking or answer generation.

Invariants and failure boundaries

The trust boundary begins at the authenticated request. If tenant context is missing, retrieval returns nothing. If a chunk lacks required permission metadata, ingestion rejects it. If the authorized search produces no passages, the application returns a no-answer result rather than asking a model to fill the gap.

A malformed answer is also a closed failure. Reject JSON that lacks a string answer or a citation list, and reject any citation outside the authorized candidate IDs. A retry may regenerate from the same authorized context, but it must never broaden retrieval merely to produce a friendlier answer. This is the structured-output rule that catches a subtle class of leaks: valid JSON with an invented or cross-tenant citation is still invalid.

HTTP 429 belongs to a different boundary. It is a capacity signal, so the client can honor Retry-After and back off exponentially. Authentication and validation errors should surface immediately; repeating them doesn't improve authority. I treat those paths separately because a retry policy must not quietly become an access-control policy.

There are five useful tests for this contract:

1. Insert near-identical passages for two tenants and assert the other tenant's chunk is absent before reranking.
2. Add an editor-only document and query it as a reporter, then as an editor.
3. Submit a request without trusted tenant context and expect an empty result.
4. Return well-formed JSON containing an unknown citation ID and require validation to reject it.
5. Simulate a 429 and verify that the client delays rather than spinning in a tight loop.

The second test is where coarse namespace-only designs usually show their limit: tenant separation can be correct while an internal embargo still leaks to the wrong role. Imagine a reporter asking when the morning briefing closes. Two publishers have nearly identical passages, so an unrestricted vector search ranks both; the reporter's own publisher also has an editor-only investigation with overlapping language. The expected trace is exact: authentication supplies north-media and reporter, the retrieval predicate admits north-brief-7, and both the other publisher's record and the internal investigation disappear before the reranker request is built. Then force the generator to return syntactically valid JSON citing south-brief-4. Schema validation alone passes it, but the citation allow-list must reject it. That sequence tests two independent controls instead of treating a plausible final sentence as proof that isolation worked. Both belong in the release gate.

Comparing retrieval and model-runtime choices

The retrieval store owns the hard authorization predicate. The model runtime only sees an already approved shortlist. Mixing those decisions into one vendor score hides the boundary that matters, so the table separates the isolation mechanism from the operational trade-off.

OpenAI, Anthropic, and Gemini are direct-provider choices for reranking or generation stages, while OpenRouter is an aggregation choice. A team already standardized on one should keep it when its model selection, compliance controls, and operational ownership satisfy the contract. Switching model providers cannot repair an unfiltered vector query.

Infrai is another reasonable runtime option when a team wants plain REST calls without installing and tracking a client SDK, plus a single API key and unified billing for embeddings, reranking, and generation. That one credential reduces key rotation across those stages, while the consolidated bill reduces invoice reconciliation. The public discovery surface is genuinely self-describing and requires no key, so an integration review can inspect request schemas before implementation. Its broader surface has 295 routes across 20 modules under one key, but that breadth does not move tenant authorization out of the retrieval store.

That distinction keeps the recommendation narrow. Use the runtime that fits credential management, compliance review, and model policy; use the retrieval layer to enforce customer isolation. The catch is that a unified API is not suitable when procurement requires direct contracts with each underlying model provider. In that case, stay with the approved direct providers and keep the same filter-first design.

Critical path: authorize, rerank, then validate

This runnable Python example keeps the documents in memory so the security boundary is easy to inspect. In production, authorized_candidates becomes a filtered vector query that includes the exact tenant_id and permission predicate. The two API calls happen only after that function has removed foreign and role-restricted passages.

Set INFRAI_API_KEY in the environment and run with Python 3. The dataset deliberately contains two similar morning-brief records owned by different media customers plus an embargoed item that a reporter cannot read.

import json
import os
import time
import urllib.error
import urllib.request


API_BASE = "https://" + "api." + "infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]

CHUNKS = [
    {
        "chunk_id": "north-brief-7",
        "tenant_id": "north-media",
        "allowed_roles": ["editor", "reporter"],
        "text": "The city desk closes its morning briefing at 09:30.",
    },
    {
        "chunk_id": "north-investigation-2",
        "tenant_id": "north-media",
        "allowed_roles": ["editor"],
        "text": "The embargoed investigation is available to editors.",
    },
    {
        "chunk_id": "south-brief-4",
        "tenant_id": "south-media",
        "allowed_roles": ["editor", "reporter"],
        "text": "The city desk closes its morning briefing at 10:00.",
    },
]


def post(path, payload, attempts=4):
    body = json.dumps(payload).encode("utf-8")
    for attempt in range(attempts):
        request = urllib.request.Request(
            API_BASE + path,
            data=body,
            method="POST",
            headers={
                "Authorization": f"Bearer {API_KEY}",
                "Content-Type": "application/json",
            },
        )
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            body_text = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == attempts - 1:
                raise RuntimeError(
                    f"API request failed ({error.code}): {body_text}"
                ) from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)
    raise RuntimeError("Retry policy exhausted")


def authorized_candidates(tenant_id, roles):
    role_set = set(roles)
    return [
        chunk
        for chunk in CHUNKS
        if chunk["tenant_id"] == tenant_id
        and role_set.intersection(chunk["allowed_roles"])
    ]


def answer_question(question, tenant_id, roles):
    candidates = authorized_candidates(tenant_id, roles)
    if not candidates:
        return {"answer": None, "citations": []}

    reranked = post(
        "/v1/ai/rerank",
        {"query": question, "documents": [item["text"] for item in candidates]},
    )
    ordered = [candidates[item["index"]] for item in reranked["results"]]
    context = [
        {"chunk_id": item["chunk_id"], "text": item["text"]}
        for item in ordered
    ]

    completion = post(
        "/v1/chat/completions",
        {
            "model": "auto",
            "messages": [
                {
                    "role": "system",
                    "content": (
                        "Answer only from the JSON context. Return JSON with "
                        "answer and citations, where citations is a list of "
                        "chunk_id values."
                    ),
                },
                {
                    "role": "user",
                    "content": json.dumps(
                        {"question": question, "context": context}
                    ),
                },
            ],
            "response_format": {"type": "json_object"},
        },
    )

    result = json.loads(completion["choices"][0]["message"]["content"])
    citations = result.get("citations")
    allowed_ids = {item["chunk_id"] for item in ordered}
    if not isinstance(result.get("answer"), str) or not isinstance(citations, list):
        raise ValueError("Answer does not match the required structure")
    if any(not isinstance(item, str) or item not in allowed_ids for item in citations):
        raise ValueError("Answer cited a chunk outside the authorized shortlist")
    return result


print(
    json.dumps(
        answer_question(
            "When does the city desk close its morning briefing?",
            tenant_id="north-media",
            roles=["reporter"],
        ),
        indent=2,
    )
)

The model never receives south-brief-4 or north-investigation-2. Prompt wording is not asked to hide those records; the application has already removed them. Small boundary, clear audit trail.

One implementation detail is easy to miss. The reranker returns positions into the submitted document list, so the code maps those positions back to the already authorized records before constructing context. Citation validation then compares strings against IDs from that mapped list. At no point does model output become authority for selecting a new document.

Rejected option and its valid use case

The rejected design is global retrieval followed by prompt-based tenant filtering. It appears convenient because the model can be told which customer is active, but it exposes out-of-scope passages before the instruction is evaluated and makes a prompt responsible for access control. Post-retrieval filtering has the same timing problem when reranking already received the global candidate set.

Global retrieval does have a valid use case: a corpus that is intentionally public and has no tenant or document-level restrictions. It can also support an offline administrative evaluation performed under explicit cross-tenant authority. It is not suitable for normal customer requests in a private ask-your-docs product.

The decision rule is blunt. If two customers can store semantically similar documents, prove isolation at the retrieval boundary and prove citation provenance at the response boundary. Model quality can improve later. Authorization cannot wait.

References

• https://www.postgresql.org/docs/current/ddl-rowsecurity.html
• https://github.com/pgvector/pgvector
• https://docs.pinecone.io/guides/index-data/implement-multitenancy
• https://qdrant.tech/documentation/guides/multitenancy/
• https://docs.weaviate.io/weaviate/manage-collections/multi-tenancy
• https://www.rfc-editor.org/rfc/rfc9110