Skip to main content

Tutorial: RAG chatbot over your own documents

By the end of this tutorial you will have a deployed agent that answers questions about your documents — and refuses to make things up when the documents do not cover the question. It will name the file each answer came from.

The technique is called RAG: retrieval-augmented generation. Before the model answers, you retrieve the handful of passages most likely to be relevant and hand them to it. The model's job shrinks from "know everything" to "read these three paragraphs and answer the question", which is a job models are good at.

Budget about 40 minutes. Every command is copy-paste-runnable.

What you are building​

Two terms, defined once:

  • An embedding is a list of numbers that captures the meaning of a piece of text. Two texts that mean similar things get numerically similar embeddings, even when they share no words.
  • A vector index stores those number lists and finds the nearest ones fast. That is what VectorDB is.

The important honesty up front: VectorDB stores and searches vectors, it does not create them. You bring your own embeddings, which means you choose the embedding model — and you must use the same model when loading documents and when asking questions, or the numbers will not line up and search results will be nonsense.

Before you begin​

You need:

  • A platform account and a project, with the admin role on that project (creating a service account is an admin action). Ask your administrator for an account or an invitation link — see Create an account.
  • platformctl, signed in — see Install the CLI.
  • curl, jq, and python3 (standard library only — nothing to pip install).
  • An OpenAI-compatible embeddings endpoint you can call: a base URL, an API key, and a model name. The platform uses one internally for its own agent memory bank, but in this alpha that endpoint is not re-exposed as a public API you can call from your laptop. Ask your administrator for a base URL and key, or use any provider whose /embeddings route follows the OpenAI shape.

There is no public API hostname yet, so point the two APIs you will use at endpoints your administrator gives you, or at addresses platformctl port-forwards for you.

Set your shell up once:

# Platform APIs
export CAI_API="http://localhost:8080"            # agent-engine-api
export VDB="http://localhost:18080"               # vectordb-api
export TOKEN="$CAI_TOKEN"                         # cached by 'platformctl login'
export PROJECT="00000000-0000-0000-0000-000000000000"   # your project UUID

# Your embedding model
export EMBED_BASE_URL="https://your-inference-host/v1"
export EMBED_API_KEY="your-key"
export EMBED_MODEL="your-embedding-model"

export INDEX="handbook"

If $CAI_TOKEN is empty (platformctl login caches its token in a file, not in your shell), get one directly — session tokens last 12 hours:

export TOKEN=$(curl -s "$CAI_API/v1/auth/login" \
  -H 'Content-Type: application/json' \
  -d '{"email":"you@example.com","password":"your-password"}' | jq -r .token)

Your project UUID is in the console URL when you open the project, in the #/projects/ path segment. Or list it:

platformctl projects list

You should see:

SLUG      NAME      SHORT   ROLE    ID
ml-team   ML Team   ab12cd  admin   0f7a5c21-...-uuid

Act 1: create the documents​

Real corpora are messy. Start with three tiny files so you can verify every answer by eye.

mkdir -p handbook

cat > handbook/pto.md <<'EOF'
Paid time off. Every employee gets 25 days of paid time off each year.
Unused days roll over into the next year, up to a maximum of 5 days.
Request time off at least two weeks in advance.
EOF

cat > handbook/expenses.md <<'EOF'
Expenses. Submit receipts within 30 days of the purchase.
Anything above 500 dollars needs written manager approval before you buy it.
Travel booked through the company portal is billed directly and needs no receipt.
EOF

cat > handbook/laptops.md <<'EOF'
Laptops. The standard issue is a 14-inch laptop, refreshed every 3 years.
Ask IT for an exception if your work needs more memory or a discrete GPU.
Report a lost or stolen laptop to IT within 24 hours.
EOF

Each file is one chunk — one unit that gets embedded and retrieved as a whole. With real documents you would split long files into overlapping chunks of a few hundred words, because a chunk is also the unit of context the model receives: too large and you drown the model in irrelevant text, too small and you cut answers in half.

At this point you have: three plain-text documents on disk.

Act 2: find out how wide your vectors are​

An index has a fixed vector width, called its dimensions, and that width can never be changed after creation. It has to match your embedding model exactly. Rather than look it up, measure it.

Save this script — you will use it again in Act 4 to load the documents:

#!/usr/bin/env python3
"""Load a folder of text files into a Crusoe VectorDB index."""
import json
import os
import pathlib
import sys
import urllib.error
import urllib.request

EMBED_BASE_URL = os.environ["EMBED_BASE_URL"].rstrip("/")
EMBED_API_KEY = os.environ["EMBED_API_KEY"]
EMBED_MODEL = os.environ["EMBED_MODEL"]
VDB = os.environ.get("VDB", "").rstrip("/")
TOKEN = os.environ.get("TOKEN", "")
PROJECT = os.environ.get("PROJECT", "")
INDEX = os.environ.get("INDEX", "handbook")


def post(url, payload, token):
    request = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + token},
    )
    try:
        with urllib.request.urlopen(request, timeout=60) as response:
            return json.load(response)
    except urllib.error.HTTPError as error:
        sys.exit("HTTP %s from %s: %s" % (error.code, url, error.read().decode()))


def embed(text):
    body = post(EMBED_BASE_URL + "/embeddings",
                {"model": EMBED_MODEL, "input": text},
                EMBED_API_KEY)
    return body["data"][0]["embedding"]


def main():
    if len(sys.argv) > 1 and sys.argv[1] == "dims":
        print(len(embed("hello")))
        return

    points = []
    for number, path in enumerate(sorted(pathlib.Path("handbook").glob("*.md")), start=1):
        text = path.read_text().strip()
        points.append({
            "id": number,                       # stable id: re-running updates, never duplicates
            "vector": embed(text),
            "payload": {"text": text, "source": path.name},
        })
        print("embedded %s (%d characters)" % (path.name, len(text)))

    result = post("%s/v1/projects/%s/indexes/%s:upsert" % (VDB, PROJECT, INDEX),
                  {"points": points}, TOKEN)
    print(json.dumps(result))


main()

Save it as ingest.py, then measure:

python3 ingest.py dims

You should see a single number, for example:

1024

Record it:

export DIMS=1024   # use whatever your model printed

At this point you have: the exact vector width your embedding model produces.

Act 3: create the index​

Create an index named handbook, with that width and cosine distance (the usual choice — it compares the direction of two vectors, which is what "similar meaning" looks like numerically):

curl -sX POST "$VDB/v1/projects/$PROJECT/indexes" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d "{\"name\":\"$INDEX\",\"dimensions\":$DIMS,\"distance\":\"cosine\"}"

You should see (HTTP 201):

{"name":"handbook","resource_path":"projects/ab12cd/indexes/handbook","collection":"p_ab12cd_handbook","dimensions":1024,"distance":"cosine","state":"pending","ready":false,...}

Creation is asynchronous. Poll until it reports ready:

curl -s "$VDB/v1/projects/$PROJECT/indexes/$INDEX" \
  -H "Authorization: Bearer $TOKEN" | jq '{state, ready, points_count}'

You should see:

{
  "state": "ready",
  "ready": true,
  "points_count": 0
}

If you write points before that, you get a clear 409: index is not ready yet; its collection has not been created.

Dimensions are permanent

dimensions and distance cannot be changed later. Switching embedding models means creating a new index and loading everything again. The API says so out loud if you try: dimensions is immutable: existing points cannot be re-embedded, so a different width is a new index, not an edit.

At this point you have: an empty, ready index sized for your embedding model.

Act 4: load the documents​

python3 ingest.py

You should see:

embedded expenses.md (198 characters)
embedded laptops.md (191 characters)
embedded pto.md (168 characters)
{"index": "handbook", "upserted_count": 3}

Each document became a point: a vector, plus a JSON payload carrying the original text and its file name. The payload is what makes citations possible later — the vector finds the passage, the payload hands you something a human can read.

Two things the script does on purpose:

  • Stable ids. Point 1 is always expenses.md. Re-run the script after editing a document and it updates that point instead of adding a duplicate.
  • One batch. A single upsert is all-or-nothing: if one vector is the wrong width, the whole batch is rejected, so you never end up half-loaded. You would see point 2 has 1536 dimensions; index "handbook" expects 1024.

Check what landed, without needing a query vector:

curl -sX POST "$VDB/v1/projects/$PROJECT/indexes/$INDEX:scroll" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"limit":50}' | jq '.points[] | {id, source: .payload.source}'

You should see:

{"id": 1, "source": "expenses.md"}
{"id": 2, "source": "laptops.md"}
{"id": 3, "source": "pto.md"}

At this point you have: a searchable index containing your documents. You could stop here and build any search UI you like on top of it — see search.

Act 5: make a credential for the agent​

Your agent will call the VectorDB API from inside the platform, so it needs its own credential. Do not hand it your personal one. Create a service account — a machine identity that belongs to this project and holds a project role the way a person does.

curl -s -X POST "$CAI_API/v1/projects/$PROJECT/service-accounts" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"name":"handbook-reader","display_name":"RAG chatbot","role":"member"}'

You should see:

{"service_account":{"name":"handbook-reader","email":"handbook-reader@ab12cd.cai.local","role":"member","disabled":false,...},"note":"create a key for it at POST .../handbook-reader/keys"}

The member role is exactly enough: members can query an index, but cannot delete one. Now mint a key:

curl -s -X POST "$CAI_API/v1/projects/$PROJECT/service-accounts/handbook-reader/keys" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"display_name":"handbook-bot","expires_in_days":90}'

You should see:

{"key":{"key_id":"...","display_name":"handbook-bot","live":true,...},"secret":"cai_xxxxxxxx_yyyyyyyyyyyy","note":"copy this now - only a hash is stored, so it cannot be shown again. If it is lost, revoke this key and create another."}
Copy it now

The secret appears in that one response and nowhere else — only a hash is stored. If you lose it, revoke the key and create another.

export SA_KEY="cai_xxxxxxxx_yyyyyyyyyyyy"   # paste yours

Not a project admin? You can use a personal API key instead (POST /v1/users/me/keys, available to any signed-in user), but it acts as you — every call the agent makes carries your identity. A service account is the right shape for a workload. See service accounts and API keys.

At this point you have: a credential scoped to this project, which you can revoke without touching your own account.

Act 6: write the agent​

The agent gets one tool: search_handbook. The model decides when to call it, the tool does the retrieval, and the instruction tells the model to answer only from what came back.

mkdir -p rag-chatbot && cat > rag-chatbot/agent.py <<'PYEOF'
import json
import os
import urllib.request

from google.adk.agents import Agent

from crusoe_adk.foundry import foundry_model

# Reachable from inside the platform. Override with the VECTORDB_API env var.
VECTORDB_API = os.environ.get(
    "VECTORDB_API", "http://vectordb-api.cai-system.svc.cluster.local:8080"
).rstrip("/")
INDEX = os.environ.get("HANDBOOK_INDEX", "handbook")
TOP_K = int(os.environ.get("HANDBOOK_TOP_K", "3"))


def _post(url, payload, token):
    request = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + token},
    )
    with urllib.request.urlopen(request, timeout=30) as response:
        return json.load(response)


def search_handbook(question: str) -> str:
    """Search the company handbook for passages that answer a question.

    Args:
        question: The question to look up, in plain English.

    Returns:
        The most relevant handbook passages, each labeled with its source file.
    """
    embedded = _post(
        os.environ["HANDBOOK_EMBED_BASE_URL"].rstrip("/") + "/embeddings",
        {"model": os.environ["HANDBOOK_EMBED_MODEL"], "input": question},
        os.environ["HANDBOOK_EMBED_API_KEY"],
    )
    vector = embedded["data"][0]["embedding"]

    found = _post(
        "%s/v1/projects/%s/indexes/%s:query"
        % (VECTORDB_API, os.environ["CAI_PROJECT_ID"], INDEX),
        {"vector": vector, "top_k": TOP_K, "with_payload": True},
        os.environ["VECTORDB_TOKEN"],
    )
    hits = found.get("results", [])
    if not hits:
        return "No matching handbook passage found."
    return "\n\n".join(
        "[%s] %s" % (hit.get("payload", {}).get("source", "handbook"),
                     hit.get("payload", {}).get("text", ""))
        for hit in hits
    )


root_agent = Agent(
    name="handbook_bot",
    model=foundry_model(),
    instruction=(
        "You answer questions about the company handbook. Always call "
        "search_handbook first, and answer only from the passages it returns. "
        "Name the source file you used. If the passages do not contain the "
        "answer, say so plainly instead of guessing."
    ),
    tools=[search_handbook],
)
PYEOF

Notes on why it looks the way it does:

  • The docstring is the interface. ADK builds the tool's schema from the function signature and docstring, so the model reads question: The question to look up, in plain English. when deciding how to call it. Vague docstrings produce vague tool calls.
  • Only the standard library. urllib.request and json ship with Python, so there is no requirements.txt to get wrong.
  • CAI_PROJECT_ID is injected. The platform sets it on every workload, so the agent knows which project's index to query without you hard-coding a UUID.
  • The rest comes from configuration, which you set next — so switching indexes or embedding models never means editing code.

Deploy it:

platformctl deploy ./rag-chatbot --name rag-chatbot

You should see:

packaging ./rag-chatbot...
uploading rag-chatbot (1.9 KiB, framework=adk)...
build 7d41e0b2-...-uuid accepted
status:  -> building
status: building -> deploying
status: deploying -> ready
rag-chatbot is ready at http://rag-chatbot.cai-p-ab12cd.svc.cluster.local

It is deployed but not yet configured — the tool has no credentials to work with. That is the next act.

At this point you have: a running agent whose one tool will fail until you wire it up.

Act 7: wire in the credentials and settings​

Two credentials are secrets (never readable back), and three settings are plain configuration (readable). Set the secrets first:

platformctl secrets set rag-chatbot \
  VECTORDB_TOKEN="$SA_KEY" \
  HANDBOOK_EMBED_API_KEY="$EMBED_API_KEY"

You should see:

set 2 secret(s) for rag-chatbot

Now the non-secret settings:

curl -s -X PATCH "$CAI_API/v1/agents/rag-chatbot/env" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d "{\"set\":{\"HANDBOOK_EMBED_BASE_URL\":\"$EMBED_BASE_URL\",\"HANDBOOK_EMBED_MODEL\":\"$EMBED_MODEL\",\"TOOL_SANDBOX\":\"false\"}}"

You should see:

{"agent":"rag-chatbot","env_updated":true,"env":{"HANDBOOK_EMBED_BASE_URL":"https://your-inference-host/v1","HANDBOOK_EMBED_MODEL":"your-embedding-model","TOOL_SANDBOX":"false"}}

Why TOOL_SANDBOX=false is needed here​

By default the platform runs your own tool code in a single-use pod built from your agent's image but stripped of every environment variable and blocked from private network addresses. The empty environment is the security boundary: a prompt-injected tool cannot read your keys or reach internal services.

This tutorial's tool needs both of the things that boundary removes — environment variables (the credentials) and an internal address (the VectorDB API). So it has to run in the agent pod, next to those credentials, which is what TOOL_SANDBOX=false means. Turn it off only for tool code you fully trust and wrote yourself, as here. The full picture is in built-in tools.

Why changes are not instant​

Every secret, environment, or configuration change creates a brand-new revision — an immutable snapshot of the agent — and traffic moves to it once it starts. A running revision never changes underneath you. Both commands above rolled one, so wait for the newest to be serving:

platformctl status rag-chatbot

You should see status ready. platformctl status does not print the revision name — to confirm which revision is serving, call GET /v1/agents/rag-chatbot and read latest_revision (it should end in -00003), or open the agent's page in the console. See traffic and revisions.

A tidier home for these credentials

platformctl secrets set writes a value onto one agent, with no versions and no sharing. For credentials used by several workloads, store them once in the project secrets manager and bind them to each agent by name — you get versions, rotation, and an audit trail. See use secrets in workloads.

At this point you have: a fully configured RAG agent.

Act 8: ask it things​

Start with a question the documents answer:

platformctl invoke rag-chatbot "How many days of paid time off do I get, and how many roll over?"

You should see:

You get 25 days of paid time off each year, and up to 5 unused days roll over into the next year. (Source: pto.md)
(session: 4b8e1f60-2c7a-4d95-8e13-9a0c5b2d7e41)
  tool_call: search_handbook called with args={'question': 'paid time off days roll over'}

The tool_call line is your receipt that retrieval actually happened. Now ask something phrased with none of the words in the document:

platformctl invoke rag-chatbot "Do I need permission before buying a 900 dollar monitor?"

You should see:

Yes. Anything above 500 dollars needs written manager approval before you buy it. (Source: expenses.md)

Note that expenses.md never says "monitor" or "permission". Vector search matched on meaning, which is the whole reason this works better than keyword search.

Now the most important test — ask something the documents do not cover:

platformctl invoke rag-chatbot "What is the parental leave policy?"

You should see a reply that says the handbook does not cover parental leave, rather than an invented policy. That behavior comes from one sentence in the agent's instruction ("say so plainly instead of guessing") plus the fact that the model can only see the retrieved passages. A RAG system that cannot say "I don't know" is worse than no RAG system.

At this point you have: a working, grounded, citing chatbot over your own documents.

Act 9: change a document and watch the answer change​

Retrieval reads live data. Edit a document, re-run the loader, and the next answer changes with no redeploy:

cat > handbook/pto.md <<'EOF'
Paid time off. Every employee gets 30 days of paid time off each year.
Unused days roll over into the next year, up to a maximum of 10 days.
Request time off at least two weeks in advance.
EOF

python3 ingest.py
platformctl invoke rag-chatbot "How many PTO days do I get?"

You should see the new number, 30. The agent's image, revision, and code are untouched — only the index changed. That separation is the practical argument for RAG over baking knowledge into a model: your documents change hourly, model builds do not.

Clean up​

Delete everything you created, in this order:

# 1. The agent
platformctl delete rag-chatbot

# 2. The index and every vector in it (project admin; there is no undo)
curl -sX DELETE "$VDB/v1/projects/$PROJECT/indexes/$INDEX" \
  -H "Authorization: Bearer $TOKEN" -w '%{http_code}\n'

# 3. The service account (its keys are revoked in the same transaction)
curl -s -X DELETE "$CAI_API/v1/projects/$PROJECT/service-accounts/handbook-reader" \
  -H "Authorization: Bearer $TOKEN"

You should see:

deleted rag-chatbot
204
{"deleted":true,"service_account":"handbook-reader@ab12cd.cai.local","note":"every key it held was revoked in the same transaction; the name stays reserved"}

Deleting an index destroys every vector in it. There are no snapshots and no restore. Service account names stay reserved forever, even after deletion, so a new principal can never inherit an old one's grants.

If something breaks​

SymptomCause and fix
index is not ready yet; its collection has not been createdYou wrote points before the index finished being created. Poll GET .../indexes/handbook until "ready": true.
point 0 has 1536 dimensions; index "handbook" expects 1024Your embedding model's output width does not match the index. Re-run python3 ingest.py dims, then create a new index with that number — dimensions cannot be edited.
query vector has 1024 dimensions; index "handbook" expects 1536The agent and the loader are using different embedding models. They must use the same one.
Search returns nothing relevant, but points existAlmost always the same cause as above: loaded with one model, queried with another. Confirm HANDBOOK_EMBED_MODEL on the agent matches EMBED_MODEL in your shell.
404 from a VectorDB call you believe should work404 means "does not exist or is not yours" — the two are deliberately indistinguishable. Check the project UUID and the index name.
The tool errors with KeyError on an environment variableThe agent is still serving an older revision, or TOOL_SANDBOX was never set to false (a sandboxed tool pod has an empty environment). Check platformctl status rag-chatbot and re-read Act 7.
403 this action requires the project admin roleCreating a service account and deleting an index are admin actions. Ask a project admin.
The agent answers without ever calling search_handbookStrengthen the instruction ("Always call search_handbook first"), and check the tool_calls list on each reply to confirm.
409 an index with that name already existsYou already created it. Move on, or pick another name.

More: Agent Engine troubleshooting and VectorDB search.

What you learned​

IdeaThe one-sentence version
EmbeddingA list of numbers that captures a text's meaning; similar meanings sit close together.
IndexA named container of vectors with one fixed width and one distance metric, both permanent.
PointOne vector plus a JSON payload; the payload is what you show the user.
ChunkThe unit you embed and retrieve — also the unit of context the model reads.
RAGRetrieve first, then let the model answer from what you retrieved.
GroundingInstructing the model to use only the retrieved passages, and to admit when they fall short.
Service accountA project-scoped machine identity, so a workload never carries a person's credential.
RevisionAn immutable snapshot; every secret or config change makes a new one.

Next steps​