Visual Search on pgvector + Gemini: Detect, Crop, Embed, Rank (and the Averaging Bug That Broke Exact Match)

I run a fashion marketplace in Azerbaijan (solo founder + AI pair programming). Users upload a photo — a screenshot from Instagram, a shop window, an outfit — and get matching products from our catalog.

Last week I gave it the cruelest test I could design: I uploaded a screenshot of a product's own catalog photo. The system had this exact image in its database.

It missed. Score: 0.9259. Exact-match threshold: 0.93. And a blue dress (0.8165) ranked above the identical orange one (0.9259).

Both failures turned out to be textbook embedding-system mistakes that I suspect half the "add vector search in a weekend" tutorials out there quietly share. Here's the full pipeline — Django, PostgreSQL + pgvector, Gemini (flash-lite gate + gemini-embedding-2) — with the real fixes.

Live demo first, if you prefer:

1. The gate: one small-model call that structures everything

Every upload hits a Gemini Flash-Lite class model once. Not "is this a product?" — that's a waste of a multimodal call. We ask for everything ranking will need, as JSON:

{
  "is_product": true,
  "brand": "",
  "items": [
    {
      "name_az": "Narıncı don", "name_ru": "Оранжевое платье",
      "color_az": "Narıncı",
      "category_public_id": "cat_...", "subcategory_public_id": "sub_...",
      "variant_item_public_ids": ["vri_..."],
      "bbox": [120, 260, 890, 640],
      "partial": false,
      "primary": true
    }
  ]
}

Three decisions that mattered:

• The gate reads our catalog. We inject a ~3K-token digest of the live taxonomy (categories, subcategories, variant vocabulary — Redis-cached 12h) into the prompt. The model returns our IDs, not free text. No fuzzy string-matching layer between detection and ranking, and item labels come back in all 4 UI languages for free.
• primary beats "largest bbox". The model marks the star of the photo. A model holding a white bag → the bag is primary, not the bigger blouse behind it.
• The ≥90% visibility rule. A trouser leg entering the frame corner is not a searchable item. Mostly-cut-off detections get partial: true — shown as a label, never cropped, never searched. This single rule killed a whole class of garbage results.

Fail-open everywhere: if the gate call dies, search continues without hints. Cost with the catalog digest: ~$0.002/search.

2. Crop + embed — twice

The primary bbox is cropped (5% padding, skip if it covers >85% of the frame — already tight). Then the asymmetry problem: catalog vectors come from full product photos; a cropped query is a zoomed fragment. Same dress, drifted fingerprints.

Cheapest fix in this article — embed both, search both, best score wins per product:

crop = crop_item_bytes(image_bytes, primary["bbox"])   # None if bbox > 85% of frame
vec_crop = embed(crop or image_bytes)
vec_full = embed(image_bytes) if crop else None

candidates = knn(vec_crop, pool=40)
if vec_full:
    candidates = merge_max_by_product(candidates, knn(vec_full, pool=40))

gemini-embedding-2 is $0.00012/image. The second embedding is insurance that costs a hundredth of a cent and rescues every catalog-screenshot query.

3. The averaging bug (read this section if you skim everything else)

Original design, the one every tutorial teaches: product has N photos → embed each → store the mean as the product vector.

What the mean actually is: a point somewhat close to every view and identical to none. My test dress had front + back photos. Uploading the back photo produced a query sitting exactly on "back photo" — measuring 0.9259 against the stored midpoint. Threshold 0.93. An identical image, diluted below my own exact bar by my own storage scheme.

And it fails intermittently: 1-photo products match perfectly, 4-photo products miss constantly. Users read that as "flaky," which is worse than "broken."

The fix — stop discarding per-photo vectors:

class VisualImageEmbedding(models.Model):
    product = models.ForeignKey(Product, on_delete=models.CASCADE)
    color_combination = models.ForeignKey(ColorCombination, null=True, ...)
    source = models.CharField(...)      # main | gallery | cc
    image_ref = models.CharField(...)   # storage path = natural dedupe key
    embedding = VectorField(dimensions=1536)

    class Meta:
        constraints = [models.UniqueConstraint(
            fields=["product", "image_ref"], name="uniq_vie_product_image_ref")]

Every photo is a row. Query-time kNN runs over views; a product's score = its best-matching view (max-merge, same pattern as the colorway vectors). The mean still gets derived — free, since the per-view vectors are already in hand — because category centroids and the PDP "similar products" rail want a prototype. Retrieval doesn't.

Re-embedding the catalog: one background job, 906 products + 301 colorways, 0 failures, ≈ $0.25 total. The row upsert checks (product, image_ref, model_id) first, so re-runs reuse stored vectors instead of calling the API again.

Post-fix, same cruel test: 0.9743. Exact, rank 1.

Rule of thumb: mean embeddings are for classification prototypes. If a user can query with any single view of an entity, store per-view vectors and max-merge. The tidy average is where your exact matches go to die.

4. pgvector notes that saved us

• Plain Cloud SQL Postgres + pgvector, HNSW indexes, same instance as the shop. No second datastore to operate.
• hnsw.iterative_scan matters: with a status='Published' filter, vanilla HNSW under-fills (asks for 40, returns 7). Iterative scan re-probes until the limit fills. We wrap it in a context manager that does SET LOCAL inside a transaction — PgBouncer-safe.
• Guarded migrations: the HNSW index is created inside RunPython with a connection.vendor == "postgresql" check, so SQLite dev/test environments don't explode.
• Per-view rows tripled the row count and changed nothing about latency worth graphing — fetch pool*3 view-rows, merge to distinct products in Python.

5. Ranking: centroids, a trust gate, and one golden rule

Cosine similarity is category-blind — it will rank a manicure set next to a face cream because both are "small object on white background." So candidates pass a reasoning stage:

Image centroids for classification. Every category/subcategory/gender keeps a centroid = mean of its product-image embeddings (averages are right here — we want prototypes). The query classifies against ~70 centroids in one pass. Leave-one-out over 583 production products: 84.7% category, 95.9% gender — ~7 points over the neighbor-vote baseline. Fun fact: our first classifier compared the image against embedded category names. Image-to-text cosine was so flat (~0.67 for right AND wrong answers) it couldn't separate hoodies from lace pajamas. Compare images with images.

The centroid build is deliberately boring — a weekly job, pure averaging, no training loop:

def build_centroids(min_samples=3):
    sums, counts = {}, {}
    for row in ProductEmbedding.objects.filter(product__status="Published"):
        for key in keys_for(row.product):          # ("category", cat_id),
            acc = sums.setdefault(key, [0.0] * DIMS)   # ("subcategory", sub_id),
            for i, v in enumerate(row.embedding):      # ("gender", g) — unisex excluded
                acc[i] += float(v)
            counts[key] = counts.get(key, 0) + 1

    for key, total in sums.items():
        if counts[key] < min_samples:      # sparse label → no centroid → falls back to vote
            continue
        VisualCentroid.objects.update_or_create(
            kind=key[0], ref_id=key[1],
            defaults={"embedding": [v / counts[key] for v in total],
                      "sample_count": counts[key]},
        )

New category added by the content team? Next weekly run picks it up. Zero maintenance, zero drift meetings. Note the irony: the classifier is built from averages — the same operation that broke retrieval. Averages are prototypes; prototypes are exactly what classification wants. Context is everything.

Trust gate: the centroid (or the gate's taxonomy) only hard-filters when confident (score + top1−top2 margin) and the strongest visual match agrees. Otherwise: plain weighted majority vote of the top candidates.

The golden rule, written in blood twice:

# nothing above the exact threshold may EVER be filtered out
pool = [c for c in pool
        if c["category"] in dominant or c["score"] >= exact_threshold]

A bikini-set photo taught us this: the gate saw "bikini top" as primary, the vote locked onto dresses, and the identical set — top score in the pool — got voted off the main grid into a side section. Exact-tier candidates are now exempt from the category vote, the taxonomy scope, color reordering, minority demotion — everything. A 93%+ match outranks every committee.

Also in this stage: color-match-first ordering (crediting the product's own color metadata, not just colorway-photo matches — the second bug from the intro), minority-subcategory demotion to a capped tail, and honest empty results. When there's no exact match, the header names the dominant subcategory of what's actually shown ("closest Dresses") — computed from the final ranked list, not from an upstream guess that ranking may have overruled.

6. Observability or it didn't happen

One structured log line per search:

[VS-RANK] hint=clothing src=gate rag_sub=48(Dresses) sub_score=0.845
          margin=0.019 gender=women(0.83) mode=focused demoted=0
          | n=12 | 19/48:0.8165, 19/48:0.9259, ...

Every incident in this article was diagnosed from these lines in minutes, not hours. Two boring-but-vital details: log the max score as top_score (we briefly logged the post-sort first element and silently poisoned two weeks of threshold-calibration data), and never log dynamic-keyed dicts to a BigQuery sink — json.dumps them into a single STRING column, or the auto-schema will mint a new column per key and jam the pipeline.

7. Failure modes: the boring table that makes it production

Three model calls, a cache and a queue sit in this request path. Every row below is implemented, not aspirational — and two of them are postmortems:

Why not CLIP / a vector DB / Google's managed visual search?

Asked constantly, so, honestly:

• Vertex AI Search for Commerce: ~$2.50/1k queries ≈ 4-6× our all-in cost, plus catalog sync, plus a ranker that can't learn our exemption rule. No.
• Vision Product Search: maintenance mode per Google's own docs. You don't build on sunset infrastructure.
• Dedicated vector DB: at ~10³ products, pgvector latency is indistinguishable and marginal ops is zero. One database, operated honestly.
• Self-hosted CLIP-family: GPU serving + a second embedding space divorced from our text search and chatbot (which share this one). The single shared space is what makes "this bag, but in brown" a future merge instead of a migration.

Buy undifferentiated capability; build what encodes your catalog's truth (the gate prompt, the centroids, the exemption rules).

8. The human loop

Every visual search lands in a moderator queue: confirm/correct the AI's category, subcategory, gender and variants (against the real variant vocabulary), and tick which shown products were actually relevant. Reviewed rows are exempt from the 90-day purge — each one is a permanent human-verified training pair. Threshold calibration reads this pool today; centroid enrichment and fine-tuning read it next. The moderation UI is a labeling shop wearing a moderation costume.

Numbers

• 906 products + 301 colorways re-embedded per-view for ≈$0.25, zero failures
• ~70 centroids (17 cat / 50 subcat / 3 gender), rebuilt weekly by a scheduled job
• Cruel test before → after: 0.9259 → 0.9743 (threshold 0.93)
• ~$0.002/search (gate + 2 embeddings); vector search marginal cost ≈ $0
• ≤4 detected pieces per photo, ≥90% visibility to be searchable, 15-min session, piece-searches free
• ~40 admin knobs, hot-reloaded, zero deploys — and every incident retired at least one knob

Takeaways

1. Per-view vectors + max-merge for retrieval; averages only as prototypes.
2. Embed noisy queries twice (full + detected object). $0.0001 insurance.
3. Classify images against image centroids, never label text.
4. Exact-threshold candidates are exempt from every filter. Write the rule before the incident.
5. Fail open on assistant calls, fail honest on results.
6. Log ranking decisions (scope, scores, demotions), not just outcomes — and log max(scores), not scores[0].
7. Ship the human-review queue with the feature, not after it. It's your training-data factory.

Questions about any layer — the gate prompt shape, iterative_scan, the centroid build — happy to go deeper in comments.

Search-engine prequel (4 languages, 2 alphabets, CIEDE2000 color science): see the canonical Medium article. Try the live feature at geyin.az — the camera button in the search bar.