Two-Stage Semantic Search with a Reranker

Vector search is fast and casts a wide net, but nearest-embedding order is not the same as most-relevant order: two passages can sit close to a query in embedding space while only one actually answers it. The standard fix is retrieve-then-rerank. Stage one uses cheap vector similarity to pull a handful of candidates (high recall). Stage two runs a cross-encoder reranker that reads the query and each candidate together and scores their relevance directly (high precision), then reorders.

This tutorial builds both stages entirely on FoundryDB:

Stage 1, recall — managed PostgreSQL with pgvector, embedding documents and the query with the managed bge-m3 model, then ORDER BY embedding <=> query.
Stage 2, precision — a managed bge-reranker-v2-m3 cross-encoder on a dedicated EU GPU, scoring each candidate against the query.

Both models run on FoundryDB GPUs in Helsinki through the same OpenAI-compatible inference gateway. Nothing leaves your EU resources, and there is no external embedding or reranking API in the loop.

What you provision

Piece	Product	Model / plan
Document + vector store	Managed PostgreSQL + `pgvector`	`tier-2`, `pgvector` extension
Embedding model	Managed inference	`bge-m3` (1024-dim), `gpu-l40s-1`
Reranker	Managed inference	`bge-reranker-v2-m3`, `gpu-l40s-1`

You also need one inference key (fdb-inf-…) with a token budget. The same key authenticates both the embedding and the reranker calls.

1. Provision the three services

Create a Postgres service with pgvector, and two inference services (embeddings and reranker). Both models are curated open-weight models, so model_source is curated and you accept the model license inline.

export FDB=https://api.foundrydb.com
AUTH="-u YOUR_EMAIL:YOUR_PASSWORD"

# Managed Postgres with pgvector
curl $AUTH -X POST $FDB/managed-services -H 'Content-Type: application/json' -d '{
  "name": "search-pg", "database_type": "postgresql", "version": "17",
  "plan_name": "tier-2", "zone": "se-sto1",
  "storage_size_gb": 50, "storage_tier": "maxiops",
  "extensions": ["pgvector"]
}'

# Embedding model (bge-m3)
curl $AUTH -X POST $FDB/inference-services -H 'Content-Type: application/json' -d '{
  "name": "search-embed", "plan_name": "gpu-l40s-1", "zone": "fi-hel2",
  "inference_config": {"model_id": "bge-m3", "model_source": "curated",
                       "served_model_name": "bge-m3", "license_accepted": true}
}'

# Reranker (bge-reranker-v2-m3)
curl $AUTH -X POST $FDB/inference-services -H 'Content-Type: application/json' -d '{
  "name": "search-rerank", "plan_name": "gpu-l40s-1", "zone": "fi-hel2",
  "inference_config": {"model_id": "bge-reranker-v2-m3", "model_source": "curated",
                       "served_model_name": "bge-reranker-v2-m3", "license_accepted": true}
}'

Wait for all three to reach Running. The inference services report Running only after the GPU is actually serving the model's task route (embeddings and scoring are probed during provisioning), so a Running reranker is a reranker you can call.

Each inference service exposes a dedicated endpoint, <name>-<id>.inf.foundrydb.com. Note the embedding and reranker hostnames from the service detail (endpoint_hostname), and mint an inference key:

export ORG=YOUR_ORG_ID
curl $AUTH -X POST $FDB/organizations/$ORG/inference/keys \
  -H 'Content-Type: application/json' \
  -d '{"name": "search-key", "monthly_token_limit": 1000000}'
# -> {"secret": "fdb-inf-…"}

2. Configure the client

Point the script at your Postgres DSN, the two inference endpoints, and the key:

export PG_DSN="postgresql://app_user:PASSWORD@search-pg-….foundrydb.com:5432/defaultdb?sslmode=require"
export EMBED_BASE="https://search-embed-….inf.foundrydb.com/v1"
export RERANK_BASE="https://search-rerank-….inf.foundrydb.com/v1"
export FDB_INF_KEY="fdb-inf-…"

The embedding endpoint speaks OpenAI's /v1/embeddings; the reranker speaks /v1/rerank (it also serves /v1/score). The model names carry the foundrydb_managed/ prefix.

3. The search script

search.py embeds and indexes a small corpus into pgvector, retrieves the nearest candidates for a query, then reranks them with the cross-encoder:

"""Two-stage semantic search on FoundryDB: managed Postgres + pgvector for recall
(bge-m3 embeddings), then a managed bge-reranker-v2-m3 GPU for precision.

Vector search is fast and casts a wide net; a cross-encoder reranker then scores
each candidate against the query directly and reorders them, so the most relevant
result rises to the top even when it was not the nearest vector.
"""
import os
import psycopg
import requests

PG_DSN = os.environ["PG_DSN"]
EMBED_BASE = os.environ["EMBED_BASE"]      # https://<embed-endpoint>/v1
RERANK_BASE = os.environ["RERANK_BASE"]    # https://<rerank-endpoint>/v1
KEY = os.environ["FDB_INF_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}

DOCS = [
    "The monthly cost circuit breaker stops all inference forwarding for an organization once its spend crosses the configured limit, until the next billing cycle.",
    "Each inference key carries a requests-per-minute limit and a monthly token ceiling that bound how fast and how much it can be used.",
    "Managed inference runs open-weight models on dedicated GPUs in Helsinki using the vLLM runtime.",
    "Storage for Files buckets is billed per gigabyte on measured usage, metered monthly.",
    "A fine-tuned LoRA adapter is hot-loaded into the running vLLM process with no restart.",
]


def embed(texts):
    r = requests.post(f"{EMBED_BASE}/embeddings", headers=HEADERS,
                      json={"model": "foundrydb_managed/bge-m3", "input": texts}, timeout=60)
    r.raise_for_status()
    return [d["embedding"] for d in r.json()["data"]]


def index():
    vecs = embed(DOCS)
    with psycopg.connect(PG_DSN) as conn, conn.cursor() as cur:
        cur.execute("CREATE EXTENSION IF NOT EXISTS vector")
        cur.execute("DROP TABLE IF EXISTS docs")
        cur.execute("CREATE TABLE docs (id serial PRIMARY KEY, content text, embedding vector(1024))")
        for content, v in zip(DOCS, vecs):
            cur.execute("INSERT INTO docs (content, embedding) VALUES (%s, %s)", (content, str(v)))
        conn.commit()
    print(f"indexed {len(DOCS)} documents")


def vector_search(question, k=5):
    qv = str(embed([question])[0])
    with psycopg.connect(PG_DSN) as conn, conn.cursor() as cur:
        cur.execute("SELECT content FROM docs ORDER BY embedding <=> %s LIMIT %s", (qv, k))
        return [row[0] for row in cur.fetchall()]


def rerank(question, candidates):
    r = requests.post(f"{RERANK_BASE}/rerank", headers=HEADERS,
                      json={"model": "foundrydb_managed/bge-reranker-v2-m3",
                            "query": question, "documents": candidates}, timeout=60)
    r.raise_for_status()
    results = r.json()["results"]
    # results: [{index, relevance_score}], highest score first
    ordered = sorted(results, key=lambda x: x["relevance_score"], reverse=True)
    return [(candidates[x["index"]], x["relevance_score"]) for x in ordered]


if __name__ == "__main__":
    index()
    q = "How do I stop runaway inference costs?"
    candidates = vector_search(q)
    print(f"\nQuery: {q}\n")
    print("Stage 1, vector retrieval order (nearest embedding first):")
    for i, c in enumerate(candidates, 1):
        print(f"  {i}. {c[:70]}")
    print("\nStage 2, reranked order (cross-encoder relevance):")
    for i, (c, s) in enumerate(rerank(q, candidates), 1):
        print(f"  {i}. [{s:.3f}] {c[:70]}")

Install the two dependencies and run it:

pip install "psycopg[binary]" requests
python search.py

4. What the reranker buys you

bge-m3 produces strong embeddings, so on a clear query vector retrieval already lands the right passage at the top. The reranker's contribution is a calibrated relevance score that raw vector distance never gives you. Take a query with a built-in trap, where two passages share vocabulary but only one is the answer:

Query: What happens to my requests when my organization hits its spending cap?

Stage 1, vector retrieval order (nearest embedding first):
The monthly cost circuit breaker stops all inference forwarding fo
Each inference key carries a requests-per-minute limit and a month
Storage for Files buckets is billed per gigabyte on measured usage
A fine-tuned LoRA adapter is hot-loaded into the running vLLM proc
Managed inference runs open-weight models on dedicated GPUs in Hel

Stage 2, reranked order (cross-encoder relevance):
[0.263] The monthly cost circuit breaker stops all inference forward
[0.002] Each inference key carries a requests-per-minute limit and a
[0.000] Storage for Files buckets is billed per gigabyte on measured
[0.000] Managed inference runs open-weight models on dedicated GPUs
[0.000] A fine-tuned LoRA adapter is hot-loaded into the running vLL

The query mentions "requests," so the rate-limit passage (which literally contains "requests-per-minute limit") sits at vector rank 2, a plausible-looking near-miss. The cross-encoder is not fooled by the shared vocabulary: it scores the true answer (the cost circuit breaker) at 0.263 and the lexical look-alike at 0.002. The gap is decisive, and it lets you do something vector distance cannot: threshold on relevance. A vector LIMIT k always returns k rows no matter how weak the match; a relevance floor keeps only what is actually relevant.

THRESHOLD = 0.05
kept = [(c, s) for c, s in rerank(q, candidates) if s >= THRESHOLD]

After a relevance floor of 0.05, 1 of 5 candidates survive:
  [0.263] The monthly cost circuit breaker stops all inference forwarding for an organization once its spend crosses the configured limit, until the next billing cycle.

Only the correct passage survives; the vocabulary-matching distractor is dropped. On unambiguous queries the same reranker returns a confident high score for the single relevant passage (for example, "Where does managed inference physically run?" scores the Helsinki-GPU passage around 0.90), so the same threshold cleanly separates signal from noise across queries.

This is exactly the stage you feed into a RAG prompt: retrieve broadly with pgvector, rerank for precision, then pass only the passages above your relevance floor to the chat model. See the In-Platform RAG tutorial for the generation half, served by the same managed inference gateway.

5. Clean up

Delete the three services when you are done so they stop accruing GPU-hours, and revoke the key:

curl $AUTH -X DELETE $FDB/managed-services/POSTGRES_ID
curl $AUTH -X DELETE $FDB/inference-services/EMBED_ID
curl $AUTH -X DELETE $FDB/inference-services/RERANK_ID

Recap

Recall then precision. pgvector retrieval is cheap and wide; the cross-encoder reranker is the precision pass that reorders and, crucially, scores relevance.
Score, not just order. The reranker's calibrated score lets you set a relevance floor and drop weak matches that a fixed LIMIT k would otherwise keep.
All on FoundryDB, all in the EU. bge-m3 embeddings and the bge-reranker-v2-m3 cross-encoder both run on managed EU GPUs behind one OpenAI-compatible gateway and one inference key; the documents and vectors live in managed Postgres. No external search or embedding API is involved.

What you provision​

1. Provision the three services​

2. Configure the client​

3. The search script​

4. What the reranker buys you​

5. Clean up​

Recap​