Skip to main content

Serverless troubleshooting

Every heading below is a real error message or symptom, verbatim, so you can find it by searching. All API errors arrive in the same envelope: {"error": "<message>", "request_id": "<id>"} — include the request_id when you ask your administrator for help.

First stop for anything stuck: read status.conditions on the service (console Overview tab, or GET .../services/<name>). Each condition — Ready, ServiceReady, VisibilityEnforced, RuntimeClassApplied, TrafficAccepted, Exposed — carries a reason and a plain-text message naming what is wrong. Also check generation vs. observed_generation: if they differ, your change was accepted but is still being applied — wait before assuming failure.

Deploying​

name must be 1-52 characters of lowercase letters, digits and dashes, start with a letter and end with a letter or digit​

Cause: the service name breaks the naming rule. The limit is 52 characters (not the usual 63) because the platform appends revision suffixes like -00001 to it.

Fix: rename: lowercase letters, digits, and dashes only; start with a letter; end with a letter or digit; 52 characters or fewer.

image is required - a serverless service has nothing to run without one​

Cause: the create request had no image field. Serverless deploys container images — there is no source-code path on this surface.

Fix: add "image": "localhost:30500/<repo>:<tag>" (or a public registry reference) to the request body.

a service with that name already exists in this project (409)​

Cause: names are unique per project.

Fix: pick another name, or PATCH the existing service if you meant to update it.

the service was modified concurrently - re-read it and retry (409)​

Cause: someone (or something) else changed the service between your read and your PATCH. The platform refuses to silently overwrite their change.

Fix: GET the service again and re-apply your edit on top of the current state.

rejected by the cluster: ... (400)​

Cause: the request passed API validation but the platform's backing store refused it; the rest of the message carries the underlying reason.

Fix: read the tail of the message — it names the exact field or rule — correct the spec, and retry.

Deploy accepted, then fails minutes later with an image pull error​

Cause: the image reference is not pullable by the machines that run your container. The most common mistake is using the in-cluster push address registry.cai-system.svc.cluster.local:5000, which only works for pushing.

Fix: reference the image via the pull prefix localhost:30500/<repo>:<tag>, or use a public registry. Then PATCH the service with the corrected image.

FailedCreate, or quota draining faster than expected​

Cause: you set a CPU or memory request without a limit. The project's LimitRange stamps a default limit of 2 CPU / 4 GiB onto your container and its networking sidecar, which can blow past project quota and surface later as FailedCreate.

Fix: set both request and limit explicitly, or omit the whole resources block and take the defaults (250m/512Mi request, 1 CPU/512Mi limit).

Quota​

SKSReady=NotReady: No Private Service Name — and nothing in the project becomes ready​

Cause: the project has hit its services quota (default 30). Every revision permanently holds 2 services, and every deploy or edit creates a revision, so the budget drains ~2 per deploy — a service edited once already holds about 4. At 30/30, new revisions across the entire project wedge in this state. Check your live number on the console's Quotas page.

Fix: free quota by deleting services you no longer need (deleting a service releases all of its revisions), then let the stuck revision retry. To avoid recurrence, keep edit churn down on busy projects and see quotas and audit — this is the sharpest quota interaction on the platform, and it is by design that a deploy only creates a revision when something actually changed.

Traffic and revisions​

traffic percentages sum to N, not 100​

Cause: :set-traffic replaces the entire split every time — it never merges with the existing split — so the targets you send must add up to exactly 100.

Fix: send the complete, final split, e.g. [{"revision_name": "api-00001", "percent": 90}, {"revision_name": "api-00002", "percent": 10}].

revision X does not exist for this service​

Cause: the split names a revision that was never created (typo, or wrong number).

Fix: list real revisions with GET .../services/<name>/revisions (or the console Revisions tab) and use an exact name like checkout-api-00002.

traffic is required - send the complete split, e.g. [{"revision_name":"api-00002","percent":100}]​

Cause: the :set-traffic body was empty or missing the traffic array.

Fix: the error message is the fix — send the whole split in one array.

unknown method ':<verb>' - the supported one is ':set-traffic' (404)​

Cause: a custom verb other than :set-traffic was appended to the service path.

Fix: the only custom verb on this API is POST .../services/<name>:set-traffic.

min_scale N is greater than max_scale M; ...​

Cause: the scaling floor exceeds the ceiling — the min_scale you sent is greater than the service's max_scale, either the one you sent in the same request or the one the service already has. A PATCH keeps sub-fields you omit, so raising min_scale alone is still checked against the existing max_scale.

Fix: raise max_scale (or lower min_scale) so the floor is at or below the ceiling. If you are changing only one of them, GET the service first to see the other's current value, or send both in the same scaling block.

Publishing​

Published, but external_url stays empty and Exposed is False​

Cause: publishing is asynchronous. The platform refuses to report a public URL — and holds the service out of Ready — until https://<name>-<project-short>.apps.codyhill.dev genuinely answers over valid TLS. The gap is usually the certificate and routing still being wired up for the hostname.

Fix: wait and poll; the Exposed condition's message names the step it is waiting on. If it stays False for a long time, share the condition message and a request_id with your administrator.

Console warns spec says private or converging on the visibility badge​

Cause: the live service and your requested spec disagree — for example, you just flipped publish off and the change is still rolling out, or the live object drifted.

Fix: converging resolves on its own; give it a moment and refresh. A persistent mismatch is worth reporting.

RuntimeClassApplied is False​

Cause: you requested runtime_class_name: "gvisor", but the platform's serving layer silently stripped it (the sandboxed runtime feature is not enabled). The condition's message names the feature flag involved.

Fix: remove runtime_class_name from the spec, or ask your administrator whether the gVisor runtime is available in your environment.

Logs​

no running pods: this service is scaled to zero. Send it a request and the logs will appear here.​

Cause: not an error — the response code is 200. The service has no running instances because it scaled to zero, so there is nothing to tail.

Fix: send the service a request, then fetch logs again. To always have logs available, set min_scale: 1 — see autoscaling.

[log stream ended: <reason>] in the middle of a log follow​

Cause: the streamed connection to the instance ended — commonly because the instance was replaced or scaled away mid-stream. Note that logs tail only the newest instance.

Fix: reconnect with ?follow=true. For history beyond the newest instance, there is no aggregated log store in the alpha.

Access and API mechanics​

404 on a service you are sure exists​

Cause: by design, "this does not exist" and "this is not your project" return the same 404, so outsiders cannot probe for names. You may be pointed at the wrong project, or lack a grant on it.

Fix: confirm the project UUID in the path (platformctl projects list), and check your membership under projects and access. The path segment must be the UUID, not the project's name.

403 on DELETE​

Cause: deleting a service requires the project admin role; members can deploy and edit but not delete. This is the only 403 on the serverless API.

Fix: ask a project admin to delete it, or to grant you the admin role.

the platform database is not configured, so project access cannot be resolved (503)​

Cause: the API is up but cannot reach the platform database that stores project grants, so it cannot authorize anyone.

Fix: this is a platform-side outage, not your request. Report it to your administrator with the request_id.

page_size must be between 1 and 200​

Cause: the list call asked for more than the maximum. Over-limit values are rejected, not silently clamped.

Fix: use page_size ≤ 200 and follow next_page_token for more pages.

page_token is invalid or was issued for a different list; start from the first page​

Cause: page tokens are opaque and bound to the exact list they came from — they cannot be reused across projects or list types, and they expire with the listing.

Fix: re-issue the list request without a page_token and page forward again.

Bursts of requests suddenly failing​

Cause: the API rate-limits each caller to about 100 requests/second with a burst allowance of 200. Tight retry loops can trip it.

Fix: slow the loop; poll with backoff rather than hammering. GET /healthz is exempt, and note it returns 200 even when degraded — read its body fields, not just the status code.

Still stuck?​

Check the platform-wide troubleshooting guide and known issues, or bring the request_id from the error envelope to your administrator.