Skip to main content

Inference overview

Everything on the platform that "thinks" is one of three model calls. This page says which is which and where each one lives; the pages under it are the detail.

The three surfaces​

You want toUsePage
Reason, answer, decide, call toolsA chat modelChat models
Turn text into vectors for searchEmbeddingsEmbeddings
Put the best search result firstRerankingReranking

An agent uses all three without you wiring anything: a chat model to think, and — in the ADK runtime — embeddings and reranking behind its memory bank. You call them directly when you are building your own retrieval over your own corpus.

All three speak the OpenAI API shape, so any OpenAI client works by changing one URL.

Hosted models​

Chat models are served by Crusoe Foundry; the platform injects a working one into every agent (CHAT_MODEL, default zai/GLM-5.2). See Chat models for pinning a different one or bringing your own provider.

Embedding and reranking models are served by the platform itself:

Model IDTask TypeOutput / Capability
qwen-embeddingText EmbeddingGenerates 4096-dimensional dense vector embeddings for semantic search.
qwen-rerankerDocument RerankingScores and ranks document passages against search queries for RAG precision.

SDK & OpenAI compatibility​

Inference endpoints follow the standard OpenAI API specification. You can use official OpenAI SDKs or standard HTTP clients by pointing them to the Inference base URL:

Python SDK example​

from openai import OpenAI

client = OpenAI(
    base_url="https://inference.data.codyhill.dev/v1",
    api_key="<your-platform-api-key>"
)

# Generate text embeddings
response = client.embeddings.create(
    model="qwen-embedding",
    input="semantic search query"
)

vector = response.data[0].embedding
print(f"Vector dimensions: {len(vector)}")  # 4096

API endpoints​

1. Text Embeddings (POST /v1/embeddings)​

Generate dense vector representations for input strings.

curl -s "https://inference.data.codyhill.dev/v1/embeddings" \
  -H "Authorization: Bearer $CAI_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen-embedding", "input": "a sailboat on the water"}'

2. Document Reranking (POST /v1/rerank)​

Score document candidates against a search query to produce relevance scores.

curl -s "https://inference.data.codyhill.dev/v1/rerank" \
  -H "Authorization: Bearer $CAI_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "a boat",
    "documents": ["a bicycle", "a sailboat", "a sports car"],
    "top_n": 2
  }'

Response format:

{
  "results": [
    {"index": 1, "document": {"text": "a sailboat"}, "relevance_score": 0.98},
    {"index": 0, "document": {"text": "a bicycle"}, "relevance_score": 0.05}
  ]
}

Results come back reordered by score, so index — the position of the document in the array you sent — is how you map a result back to your own record. Never use the position in results. document.text is your original text, restored by the platform: the model server sees an internally templated form of each document and echoes that back, so the platform substitutes what you actually sent.

3. Model Catalog (GET /v1/models)​

List available models served by the Inference platform.

curl -s "https://inference.data.codyhill.dev/v1/models" \
  -H "Authorization: Bearer $CAI_TOKEN"

CLI management​

Use platformctl to query models, generate embeddings, and run reranking operations from your terminal:

# List available models
platformctl inference models

# Generate embeddings
platformctl inference embed "a sailboat on the water"

# Rerank document candidates
platformctl inference rerank "a boat" "a bicycle" "a sailboat"

Automatic agent memory integration​

Deployed AI agents utilize hosted embedding and reranking models automatically for long-term memory indexing and RAG retrieval.

  • Vector Consistency: All memories created by an agent are indexed using 4096-dimensional qwen-embedding vectors.
  • Automated reranking in agent memory: the ADK agent runtime's search_memory retrieves a wide candidate set — 4x the requested count, minimum 20 — from the memory bank and reorders it through qwen-reranker before returning the top results. This is the memory bank only: the LangGraph and CrewAI runtimes do not do it, and retrieval over a VectorDB index you built yourself never does. If the rerank call fails, the search falls back to plain vector order rather than erroring. To get the same quality over your own index, call /v1/rerank yourself — see Search.

Operational Specifications & Limits​

  • Endpoint URL: From your own machine or any client outside the platform, https://inference.data.codyhill.dev/v1, or the shared platform API https://api.codyhill.dev — /v1/embeddings, /v1/rerank and /v1/models route to Inference on both, and platformctl defaults to the latter. From inside a workload, use the address the platform injects: read EMBED_BASE_URL rather than writing a URL of your own, with EMBED_MODEL naming the model. The public hostnames do not route from inside a project — the request hangs rather than failing — so never hardcode one into an agent or function.
  • Authentication: Authenticated via platform API keys, service accounts, or session tokens.
  • Request Throttling: Edge rate limits prevent abuse and maintain predictable latency across inference workloads.

Next steps​