Skip to main content

Reranking

Vector search is fast and approximately right. A reranker is slow and precisely right. Used together — search wide, then rerank — you get both.

Why a second stage​

A vector search compares the question to each chunk as two separate embeddings that were computed without knowing about each other. That is what makes it fast enough to scan millions of points, and it is also why the third result is often better than the first.

A reranker reads the question and one document together and scores that pair. It cannot scan a million documents. It can put twenty in the right order.

So the pattern is:

  1. Retrieve wide — ask VectorDB for 20–50 candidates instead of 5.
  2. Rerank — score those candidates against the query.
  3. Keep the top few — hand 3–5 to the model.

The cost is one extra call. The benefit is that the answer is grounded in the passage that actually answers the question.

The model​

Model idTask
qwen-rerankerScores query/document pairs for relevance

Rerank a candidate set​

platformctl inference rerank "a boat" "a bicycle" "a sailboat" "a sports car"
index maps back to your array — the position in results does not

Results come back reordered by score. index is the position of that document in the array you sent, and it is the only safe way to get back to your own record — the chunk's id, its source file, its URL. Using the position within results silently attaches the wrong citation to the right answer, which is the kind of bug that survives review because every individual piece looks correct.

document.text is your original text, restored by the platform. The model server sees an internally templated form of each document and echoes that back, so what you get is what you sent.

The whole pattern, end to end​

import os
from openai import OpenAI

client = OpenAI(base_url=os.environ["EMBED_BASE_URL"], api_key=os.environ["MODEL_API_KEY"])

def answer(question: str, index_client, top_k: int = 5):
    # 1. Embed the question.
    qvec = client.embeddings.create(
        model=os.environ["EMBED_MODEL"], input=question
    ).data[0].embedding

    # 2. Retrieve WIDE. 40 candidates, not 5 - the reranker needs something to
    #    choose between, and a candidate that never gets retrieved can never be
    #    promoted.
    hits = index_client.search(vector=qvec, limit=40)

    # 3. Rerank those candidates against the question.
    docs = [h.payload["text"] for h in hits]
    ranked = rerank(question, docs, top_n=top_k)

    # 4. Map back through `index` - never through the position in `ranked`.
    return [hits[r["index"]] for r in ranked]

Choosing the numbers​

SettingA reasonable defaultWhy
Candidates retrieved20–50Enough for the reranker to work with, small enough to score quickly
top_n returned3–5What actually fits in a prompt without diluting it
Score floornone, at firstMeasure real scores on your own corpus before rejecting anything

Resist setting a minimum relevance score before you have looked at real numbers. Scores are not calibrated across corpora, and a floor copied from an example quietly returns nothing on a corpus where 0.4 is a good match.

Agents rerank their memory for you​

The ADK agent runtime does this automatically for the memory bank: search_memory retrieves a wide candidate set — 4× the requested count, minimum 20 — and reorders it through qwen-reranker before returning the top results.

Three limits worth knowing:

  • It is the memory bank only. Retrieval over a VectorDB index you built yourself never reranks unless you call this API.
  • It is the ADK runtime only. The LangGraph and CrewAI runtimes do not.
  • If the rerank call fails, the search falls back to plain vector order rather than erroring. The answer gets slightly worse; nothing breaks.

To get the same quality over your own index, do what the runtime does — the pattern above.

When not to bother​

  • Exact lookups. If the user gave you an order number, query by filter, not by similarity.
  • One or two candidates. Reranking three documents mostly buys latency.
  • Latency budgets under a second. The extra call is real. Measure it before you promise it.

Next steps​