Skip to main content

Troubleshooting

This page collects the errors you can hit anywhere on the Crusoe Agent Platform, grouped by what you were doing when you hit them. The headings are the real error strings, so searching this page for the text you saw should land you on the fix.

Each service also has its own focused page: agents, serverless services, and functions.

Before you start: how to read an error​

Every API on this platform returns errors in one shape:

{"error": "a plain-English description of what went wrong", "request_id": "8f3c1a2b"}

The request_id also comes back as an X-Request-Id response header, and the console shows it on every error toast. Quote it when you ask for help — it is how support finds the exact log lines for your call.

Two rules explain most confusing responses:

  • 404 means "not found or not yours." If you hold no grant on a project, everything under it answers 404 not found — never 403. This is deliberate: it stops anyone from discovering which projects and resources exist. You only see 403 once you already have some access but lack the role for that one action.
  • Deploys are asynchronous. Creating an agent, function, service, index, or MCP server returns 202 or 201 immediately. That means "accepted", not "running". Poll the resource until it reports ready.

Signing in​

invalid email or password​

Returned as 401 from POST /v1/auth/login and from the console login form.

Cause: the platform returns one message for both "no such account" and "wrong password", on purpose — otherwise anyone could use the login form to discover who has an account.

Fix: check the address for typos. If you are sure the password is right, ask your administrator to reset it (POST /v1/users/{email}/reset-password issues a one-time temporary password). Passwords are at least 12 characters, so a short one you remember is probably from a different system.

too many sign-in attempts; try again in 12m30s​

Returned as 429, with a Retry-After header.

Cause: sign-in is throttled at 10 attempts per IP address per 15 minutes and 50 attempts per account per 15 minutes.

Fix: wait the time the message names, then try once more. A 429 never means your password is wrong — do not keep guessing, because every attempt extends the block. If a script is retrying in a loop, stop it first.

password change required before using this API​

Returned as 403 on every management route.

Cause: your account was created by an administrator with a temporary password. You can sign in, but you cannot manage anything until you choose a new one.

Fix: sign in at https://console.codyhill.dev and set a new password (minimum 12 characters) in the modal that appears. Then run platformctl login again so the CLI caches a fresh token. See create an account.

password must be at least 12 characters / new password must differ from the current one​

Cause: the platform enforces a 12-character minimum everywhere, and refuses a "change" that changes nothing.

Fix: pick a longer, different password. Length beats complexity here — there is no symbol or digit requirement.

user management is not initialized​

Returned as 503 from login.

Cause: the platform's identity store is not up. This is not something you did.

Fix: tell your administrator. Nothing you change on your side will help.

I can't find the sign-up page​

Cause: there isn't one. This platform has no self-service sign-up.

Fix: ask your administrator for an account, or for an invitation link to a project. See create an account.

Cause: the token in the link is wrong (404) or the invitation is no longer open (410). Invitation tokens are shown once, when the invitation is created — the platform cannot retrieve one later.

Fix: ask whoever invited you to issue a new invitation. If you already accepted successfully, just sign in normally; accepting twice is not needed and never changes an existing account's password.

Permissions and access​

not found on something you are certain exists​

Cause: the 404 rule. You hold no grant on that project, so its contents are invisible to you — including their existence.

Fix:

platformctl whoami
platformctl projects list

If the project isn't listed, ask a project admin to add you (see projects and access). If it is listed but resources still 404, check you are scoped to the right one:

platformctl config set-project my-project

this is a management endpoint and requires authentication​

Full message (401):

this is a management endpoint and requires authentication. Sign in (POST /v1/auth/login) and send 'Authorization: Bearer <token>', or use the automation token: kubectl -n cai-agents get secret cai-automation-token -o jsonpath='{.data.token}' | base64 -d

Cause: everything that creates, changes, or inspects platform state needs a credential. Only invoking an agent is open by default.

Fix: platformctl login, or send Authorization: Bearer $CAI_TOKEN with an API key. See API authentication.

this action requires the project admin role​

Returned as 403.

Cause: you are a project member, and this action is admin-only. Admin-only actions include: revealing a secret, deleting a secret, deleting a serverless service, deleting or editing a VectorDB index, deleting or rotating a MemoryStore instance, creating and deleting Pub/Sub topics and subscriptions, fetching direct-access credentials, and creating service accounts and their keys.

Fix: ask a project admin to do it, or to promote you.

a service account cannot create or manage credentials.​

Full message (403):

a service account cannot create or manage credentials. Sign in as a user (or use your own API key) to issue keys - otherwise a leaked key could mint replacements and revoking it would achieve nothing

Cause: you used a service-account key to try to create another key, an account, or a membership.

Fix: do credential work as a human — sign in, or use your own personal API key. See service accounts and API keys.

this credential is a deployed workload's key; it can only mint a read token for its own project's secrets​

Cause: something inside a deployed workload used its automatically injected key on a management route. That key has exactly one power: minting a short-lived read token for its own project's secrets.

Fix: if a workload genuinely needs broader access, give it a service-account API key as a secret. See use secrets in workloads.

I'm a platform admin, but the project's pages are empty​

Cause: by design. Authority has two axes: authority over the project object (rename, delete, membership) and authority over its contents (agents, secrets, data). A platform admin holds the first everywhere and the second nowhere — provider staff cannot read tenant data.

Fix: take a break-glass grant. It is visible to the project's members, requires a written reason, expires on its own (4 hours by default, 24 hours maximum), and is recorded in the audit log. See break-glass and audit.

Deploying​

invalid agent name (must be a lowercase DNS label)​

Cause: agent and function names must match ^[a-z]([a-z0-9-]{0,61}[a-z0-9])?$ — lowercase letters, digits, and hyphens, starting with a letter, at most 63 characters. MyAgent, my_agent, and my agent are all rejected.

Related names have their own rules:

ThingRule
Agent / functionlowercase DNS label, ≤ 63 chars
Serverless servicelowercase DNS label, ≤ 52 chars (revisions append -00001)
MCP serverlowercase DNS label, ≤ 40 chars
MCP tool^[a-z][a-z0-9_]{0,62}$, no leading underscore
VectorDB indexlowercase letters, digits, hyphens, ≤ 48 chars
MemoryStore instancelowercase letters, digits, hyphens, ≤ 40 chars, starts with a letter
Pub/Sub topic and subscriptionlowercase letters, digits, dashes, ≤ 63 chars
Secretletters, digits, - or _, ≤ 63 chars

Fix: rename to something like my-agent. Names are immutable — you cannot rename a deployed resource, only delete and recreate it.

this project is at its service limit​

Full message (409):

this project is at its service limit (30 / 30): deploying needs at least one more service and cannot proceed. Delete an agent or function, or ask an admin to raise the project's service quota, then deploy.

Cause: every agent, function, and serverless service draws on your project's Services quota, and it is full.

Fix: delete something you no longer need, or ask an admin to raise the quota. Check current usage under Quotas in the console — see quotas and audit.

Serverless revisions cost more than you think

Each revision of a serverless service permanently holds two Kubernetes Services, so every deploy of the same service draws the quota down again. If a project stops becoming ready for no obvious reason, the Services quota is the first thing to check.

a build is already in progress for agent research-buddy​

Cause: two deploys of the same agent raced. Only one build runs at a time.

Fix: wait for the running build (platformctl status <name>), then deploy again. A build that crashed leaves a claim that becomes reclaimable after 30 minutes.

Status goes to failed and there is no build log​

Cause: the build failed — usually a dependency that cannot be installed.

Fix: there is no separate build-log endpoint. The tail of the build output (up to 4,000 characters) lands in the resource's message field:

platformctl status my-agent

You should see the failing step near the end of message. For LangGraph and CrewAI agents, dependency pins that conflict with the platform's base image fail here at build time, deliberately, rather than crashing at import time later.

the uploaded source archive is larger than the console accepts (32 MiB)​

Full message (413 from the console):

the uploaded source archive is larger than the console accepts (32 MiB); deploy it with `platformctl deploy` instead, which uploads to the agent engine directly

Cause: the browser proxy caps uploads at 32 MiB. The platform itself accepts 100 MiB.

Fix: use the CLI:

platformctl deploy ./my-agent --name my-agent

The console also refuses binary files and paths longer than 100 bytes in its "write" and "files" modes. Both work fine through the CLI, or through the console's ready-made .tar.gz mode.

The deploy succeeded but nothing runs, and minutes later there's an image pull error​

Cause: a serverless service was given an image reference that cluster nodes cannot pull.

Fix: reference images by the pull prefix localhost:30500/<repo>:<tag>, a public registry, or Crusoe Container Registry. The push endpoint (registry.cai-system.svc.cluster.local:5000) is not pullable by nodes. See deploy a service.

Invoking agents and functions​

unknown agent: research-buddy​

Cause: either no agent with that name exists, or one does but you cannot see it. These are deliberately the same 404.

Fix: confirm the name and project:

platformctl list --project my-project

If a teammate deployed it, ask them or a project admin.

the agent name research-buddy exists in more than one project​

Full message (409):

the agent name research-buddy exists in more than one project; add ?project=<slug> to say which one, or sign in so it resolves within your project

Fix: add ?project=my-project to the URL, or pass --project my-project on the CLI.

session_id chosen by an unauthenticated caller must be at least 24 characters​

Full message (400):

session_id chosen by an unauthenticated caller must be at least 24 characters of unguessable randomness (or omit it and the platform will generate one). A short, guessable id would let anyone else read this conversation.

Cause: you invoked anonymously and chose a short session id such as s1. Anyone who guessed it could read your conversation.

Fix: omit session_id and reuse the one the platform returns:

curl -sS -X POST "$CAI_API/v1/agents/research-buddy/invoke" \
  -H 'content-type: application/json' \
  -d '{"message":"hello"}'

You should see:

{"session_id":"5b2f...","user_id":"...","output":"...","reasoning":"","tool_calls":[],"events":[]}

Signed-in callers are exempt from the 24-character rule. See sessions.

memorize requires authentication​

Full message (401):

memorize requires authentication: it writes durable memory that later callers read back. Invoke without 'memorize', then call POST /v1/agents/{name}/sessions/{id}/memorize with a session token.

Cause: memorize writes into the agent's shared memory bank, which every later caller can search. Invoking is open; writing durable memory never is.

Fix: platformctl login, then platformctl memorize <agent> --session <id>. See memory.

Turn 2 doesn't remember turn 1​

Cause: the session_id changed between calls, or you supplied two different user_id values.

Fix: reuse the same session_id every turn (--session on the CLI). If you set user_id yourself, keep it constant — or omit it and the platform derives a stable one from the session.

The first call after a quiet period is slow, or times out​

Cause: a cold start. Idle workloads run zero instances; the first request has to start one and import your code, all inside the 60-second invoke timeout.

Fix: retry once — the second call hits a warm instance. To remove cold starts entirely on a latency-sensitive workload, set its minimum instances to 1 (the Configure modal in the console, or PATCH /v1/agents/{name}/config). That keeps one instance always running, which costs capacity and quota. See autoscaling and scale to zero.

The agent replies with a model authentication error​

Cause: no model credential is in force — the platform-wide default key is missing and the agent has none of its own.

Fix: the agent's own URL serves an unauthenticated GET /debug/config that reports model_key_present and the resolved model name without revealing any value. If it reads false, set a key on the agent:

platformctl secrets set my-agent MODEL_API_KEY=sk-...

That rolls a new revision. See secrets and env.

body exceeds 8388608 bytes when calling a function​

Cause: a function's POST body is capped at 8 MiB by the language shim.

Fix: send less, or pass a reference (an object key, a topic message) instead of the payload itself.

Running code in the sandbox​

no warm sandbox pod available, try again shortly​

Returned as 503.

Cause: every run_python call takes a fresh, single-use pod from a small warm pool, and the pool was empty for the full 30-second wait.

Fix: retry in a few seconds. If it happens constantly, you are running more concurrent sandbox calls than the pool can replenish — spread the work out. See security and limits.

timed out after 20s with exit_code: -1​

Cause: the snippet ran past its timeout and was killed. run_python uses a 20-second execution timeout; direct sandbox calls accept up to 60 seconds.

Fix: make the snippet finish faster, or raise timeout_s (maximum 60) if you are calling the sandbox directly. An infinite loop always ends this way — that is the safety net working.

Code in run_python cannot fetch a URL​

Cause: sandbox pods used for run_python have no network egress except DNS. Nothing model-generated can call out.

Fix: fetch the data in your own tool code (which runs with internet access) and pass it into the snippet, or use the agent's tools instead of network calls from generated code. See run code.

Working with data services​

index is not ready yet; its collection has not been created​

Returned as 409 from VectorDB upsert, query, scroll, or delete-points.

Cause: index creation returns 201 immediately with state: "pending"; the storage is created a moment later.

Fix: poll until ready before writing:

platformctl vectordb get docs

You should see:

NAME  DIMENSIONS  DISTANCE  STATE  READY  POINTS
docs  1536        cosine    ready  yes    0

point 3 has 1536 dimensions; index "docs" expects 256​

Cause: every vector in an index must have exactly the index's width. One bad point rejects the whole batch — there are no partial writes.

Fix: re-embed with the model that matches the index, or create a new index at the right width. dimensions is immutable.

dimensions is immutable: existing points cannot be re-embedded, so a different width is a new index, not an edit​

Cause: you tried to PATCH an immutable field. dimensions, distance, shards, on_disk, and quantization are all fixed at creation. Only replicas (1–8) and payload_on_disk can change.

Fix: create a second index with the settings you want and re-ingest your data, then delete the old one.

an empty filter would delete every point; delete the index instead if that is the intent​

Cause: you sent delete-points with an empty filter object.

Fix: name the ids, or write a filter that actually selects something. To wipe the index, delete the index.

MemoryStore refuses my password after a rotation​

Cause: rotation is not instantaneous. The server reads its password once, at start, so the new password takes effect only when the instance finishes restarting. Until then the previous password is still the one in force.

Fix: wait for the instance to report Ready again, then use the new password. Read it from the ms-<name>-credential Kubernetes Secret rather than pasting it around.

this memorystore is not provisioned yet; wait for it to report Ready, then read its stats​

Cause: you asked for live stats before provisioning finished (409), or the instance is unreachable right now (503 could not reach this memorystore to read its stats; it may be scaled down, still provisioning, restarting, or mid-credential-rotation).

Fix: poll platformctl memorystore get <name> until READY is yes, then retry.

My workload connects to the wrong MemoryStore​

Cause: the injected environment variable CRUSOE_MEMORYSTORE_ADDR points at the platform's shared store that backs agent sessions — not at any instance you created.

Fix: connect to your own instance using the address in its connection.host field (ms-<name>.<project-namespace>.svc.cluster.local, port 6379) and the password from its credential Secret. See connect from workloads.

Writes to MemoryStore start failing when the instance fills up​

Cause: the default eviction policy is noeviction, which fails writes loudly at the memory limit instead of silently dropping data. That is right for session state and wrong for a cache.

Fix: for cache workloads, create the instance with "maxmemory_policy": "allkeys-lru". The policy is set at creation and is not changed in place.

Messaging (Pub/Sub)​

this project's Pub/Sub storage budget is 1.0GiB, of which 64.0MiB is already claimed​

Full message (409):

this project's Pub/Sub storage budget is 1.0GiB, of which 1.0GiB is already claimed by 64 topic(s); a topic of 16Mi does not fit. Delete a topic, lower an existing topic's max_bytes, or ask the platform operator to raise the project's quota.

Cause: the budget is spent by claim, not by use. A topic reserves its whole max_bytes the moment it exists, so four empty 16 MiB topics have already spent 64 MiB while every byte counter reads zero.

Fix: delete an unused topic, or lower an existing topic's max_bytes. Consuming messages does not free budget.

I published a message and nothing ever received it​

Cause: a message is retained only while some subscription still owes an acknowledgement. Publishing to a topic with no subscriptions succeeds and returns a message id, but the message is reclaimed on the broker's own schedule.

Fix: create the subscription before you publish. See topics and subscriptions.

this is a push subscription; its messages are delivered to http://...​

Returned as 409 on pull.

Cause: a subscription is either pull or push, not both.

Fix: create a second, pull subscription on the same topic if you want to read messages by hand while push delivery keeps running.

this subscription is ordered (type: exclusive), so it admits one reader at a time and cannot be served by a load-balanced API​

Cause: ordered subscription types (exclusive, failover) allow exactly one active reader, which a load-balanced HTTP API cannot provide.

Fix: use push delivery for ordered types, or create a shared or key-shared subscription if you can tolerate no total ordering.

deliver.push.url is not allowed​

Full message (400):

deliver.push.url is not allowed: <reason>. A push target must be a Service in this project's own namespace (cai-p-ab12cd), e.g. http://my-worker.cai-p-ab12cd.svc.cluster.local:8080/events

Cause: push targets are restricted to services inside your own project, over plain http, with no redirects. This makes it structurally impossible to aim deliveries at someone else's workload or at the internet.

Fix: use the exact in-project form the message shows.

Messages keep coming back forever on a pull subscription​

Cause: max_deliver is enforced by the push worker only. On a pull subscription, a message you never acknowledge redelivers indefinitely.

Fix: acknowledge what you finish, and use the delivery_attempt field in the pull response to decide when to give up on a poison message. On the CLI, --ack acknowledges what it pulled:

platformctl pubsub subscriptions pull workers --topic orders --max 10 --ack

Secrets​

openai-api-key is still bound by 2 binding(s)​

Full message (409):

openai-api-key is still bound by 2 binding(s): research-buddy.OPENAI_API_KEY, ... Remove them first - nothing in the database prevents this delete, so the failure would otherwise appear later, as an agent that will not start.

Fix: unbind it from each agent listed, then delete.

no secret called "openai-api-key" in this project​

Cause: you tried to bind a secret that hasn't been created. The platform refuses now rather than letting the agent fail at deploy time.

Fix: create it first — see manage secrets.

MODEL_API_KEY is reserved by the platform and cannot be bound​

Cause: some environment variable names are set by the platform itself (the model endpoint and credential, sandbox address, messaging credential, process loader). Binding over them would either be ignored or break the workload.

Fix: choose a different environment variable name for the binding. If you specifically want to point an agent at your own model credential, set it as a direct per-agent secret instead:

platformctl secrets set my-agent MODEL_API_KEY=sk-...

nothing was written. At least one binding could not be read​

Returned as 424 from apply.

Cause: applying bindings is all-or-nothing. If any value cannot be read, nothing is written — a half-configured agent is worse than an unchanged one.

Fix: the response lists each binding and which one failed. Fix that one (usually a destroyed version, or a store that is sealed), then apply again.

the project secret store is not configured​

Cause: the platform's secret store is not switched on. A sealed store gives a different message, telling you it exists but cannot answer.

Fix: both are operator problems — tell your administrator. Note that "no secrets yet" always renders as an empty list, never as an error, so an error here really does mean the store is unavailable.

I applied a secret but the agent still doesn't see it​

Cause: revisions are immutable snapshots. A running instance never picks up a new value; a new revision has to start.

Fix: every secret write rolls a new revision automatically, but the agent only reads it at its next cold start. Invoke it once, then check:

curl -s https://my-agent-ab12cd.apps.codyhill.dev/debug/config

You should see the key listed in config_env_keys (names only — values are never returned).

Using the CLI​

this command needs a project​

Full message:

this command needs a project: pass --project, set $CAI_PROJECT, or run 'platformctl config set-project' (use the id or slug shown by 'platformctl projects list')

Cause: the serverless, pubsub, memorystore, and vectordb command groups address resources by project id and have no server-side default.

Fix:

platformctl projects list
platformctl config set-project my-project

$CAI_PROJECT overrides the saved default, and --project overrides both.

auto port-forward failed (pass --api or set CAI_API instead)​

Cause: with no endpoint configured, the CLI tries to reach the platform by opening a kubectl port-forward tunnel — which needs kubectl and a working KUBECONFIG. There is no public API hostname in the alpha.

Fix: either fix your cluster credentials, or point the CLI at an endpoint your administrator gave you:

export CAI_API=https://api.example.internal

The four service groups use their own variables: CAI_SERVERLESS_API, CAI_PUBSUB_API, CAI_MEMORYSTORE_API, CAI_VECTORDB_API. The global --api flag names the agent control plane only.

agent-engine-api returned 401: ... [platformctl credential: none; run 'platformctl login' or set $CAI_TOKEN]​

Cause: no credential was found. The bracketed note tells you which source the CLI used, in this order: $CAI_TOKEN, the cached login token, $CAI_USER plus $CAI_PASSWORD, then the cluster automation token.

Fix: platformctl login, or export an API key as $CAI_TOKEN. Sessions last 12 hours, so a cached token from yesterday is expired.

--follow and --history are mutually exclusive​

Fix: pick one. -f tails the live instance; --history reads persisted lines that survive scale-to-zero.

invalid --output "csv": want table, json, or yaml​

Fix: use -o table (default), -o json, or -o yaml.

--session is required​

Cause: platformctl memorize needs to know which conversation to commit to memory.

Fix: pass the session id that platformctl invoke printed: platformctl memorize my-agent --session 5b2f....

My deploy uploaded my whole virtualenv​

Cause: platformctl deploy archives everything in the directory. There is no ignore file.

Fix: keep agent directories clean, or deploy from a copy. A stray .venv/ or node_modules/ will be uploaded and can blow the 100 MiB cap.

My ADK agent deployed as CrewAI​

Cause: the framework is auto-detected from the entry file: crew.py means CrewAI, graph.py means LangGraph, otherwise ADK. A stray file changes the answer. Function runtimes are detected the same way from handler.js, handler.go, handler.rb.

Fix: be explicit: platformctl deploy ./my-agent --framework adk.

Lists, pagination, and rate limits​

page_size must be between 1 and 200​

Cause: every list accepts page_size between 1 and 200 (default 50). Asking for more is rejected, not silently reduced.

Fix: request 200 or fewer and follow next_page_token. The CLI does this for you.

page_token is invalid or was issued for a different list; start from the first page​

Cause: page tokens are opaque and bound to the exact list that issued them. A token from one list, project, or filter is refused by another.

Fix: drop the token and start from page one.

Sudden 429 responses under load​

Cause: every API applies a per-credential rate limit, falling back to per-IP for anonymous callers.

Fix: back off and retry. If a batch job needs steady throughput, spread it out rather than bursting, and give it its own service-account key so it doesn't share a bucket with your interactive work.

Still stuck?​

  1. Quote the request_id from the error body — it identifies your exact call.
  2. Check known issues; some rough edges in this alpha are documented rather than fixed.
  3. Check the service-specific pages: agents, serverless, functions.
  4. Unsure what a word means? See the glossary. Common questions are answered in the FAQ.