A durable agent pipeline

Agents are chatty and occasionally slow. They cold-start, they call models that rate-limit, they time out. If you chain two agent calls together in a shell script and the second one fails, you lose the first one's work and start over.

A workflow fixes that. It is ordinary code whose progress is written down after every step. A failed step retries on its own. A crashed process picks up where it left off instead of at the beginning. In this tutorial you'll run the platform's sample workflow — agent-pipeline — which invokes an agent twice, carrying the conversation from the first call into the second, then you'll watch its retry policy handle a failure and read the complete recorded history.

Budget about 20 minutes.

Read this before you start: honest alpha scope

Durable Execution is the least finished surface on the platform. It ships as a sample worker integration, not a managed service:

• No REST API, no platformctl command, no console page. Everything here is kubectl plus the temporal command-line tool, run from inside the cluster.
• The workflow engine has no public endpoint and no platform authentication. It listens in-cluster only; being on the cluster network is the entire security boundary. Your worker must run in-cluster, and you must not put secrets in workflow inputs.
• All projects share one workflow namespace. Workflow IDs are cluster-global, so prefix yours (ml-team-demo-1, not demo-1) or you'll collide with someone else.
• Every engine component runs as a single replica. A restart pauses running workflows; it doesn't lose them, but it does stall them.

Everything below genuinely works today. Just size your expectations to "a proven sample you can copy", not "a product with an SLA".

What the sample workflow does

Two terms, in plain English:

• An activity is one retryable step of a workflow. Here, each activity is a single call to an agent's invoke endpoint.
• A worker is your own process that hosts the workflow and activity code. It watches a named mailbox — the task queue — and runs whatever work shows up there. The workflow engine never runs your code; it just records what happened and tells your worker what to do next.

The sample takes three inputs: an agent name and two messages. The second call reuses the session_id the first call returned, which is what makes the agent remember the first exchange — the same mechanism described in sessions.

Every activity carries this retry policy, from the sample's workflow.go:

Setting	Value	What it means
StartToCloseTimeout	30s	One attempt gets 30 seconds to finish.
InitialInterval	1s	The first retry waits 1 second.
BackoffCoefficient	2.0	Each wait doubles: 1s, 2s, 4s...
MaximumAttempts	3	After 3 attempts, give up and fail the workflow.

You write none of that per call site. It is declared once and applies to every attempt.

Before you begin

• kubectl access to the platform cluster. There is no other way in for this surface.
• A deployed agent to invoke. This tutorial uses research-buddy from the agent quickstart.
• The sample source at examples/workflows/agent-pipeline in the platform repository, and its worker image published to the cluster registry — your platform operator's standard image build does this.

Check the agent is up first. A workflow that fails because the agent doesn't exist is a confusing first run:

platformctl status research-buddy

You should see a row reporting status ready.

1. Deploy the worker

The worker is a plain Kubernetes Deployment. It hosts the workflow and activity code and polls the agent-pipeline task queue.

kubectl apply -f examples/workflows/agent-pipeline/deploy.yaml

You should see:

deployment.apps/agent-pipeline-worker created

It runs in the cai-services namespace and connects to the workflow engine at temporal-frontend.cai-services.svc.cluster.local:7233, set through its TEMPORAL_ADDR environment variable. Confirm it came up:

kubectl -n cai-services get deploy agent-pipeline-worker

You should see:

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
agent-pipeline-worker   1/1     1            1           30s

If READY stays at 0/1, the worker image probably isn't in the registry yet — check with kubectl -n cai-services describe deploy agent-pipeline-worker and ask your operator to publish it.

2. Get a shell on the admin tools

The temporal command-line tool ships in an admin-tools pod, so you don't install anything locally:

POD=$(kubectl get pods -n cai-services \
  -l app.kubernetes.io/name=temporal-admintools \
  -o jsonpath='{.items[0].metadata.name}')
echo "$POD"

You should see a pod name like:

temporal-admintools-6d9f7c9b8d-x2k4p

3. Start the workflow

The workflow takes three positional inputs, one --input flag each, and each one is a JSON value — which is why the strings are quoted twice: the outer quotes are the shell's, the inner ones are JSON's.

kubectl exec -n cai-services "$POD" -- temporal workflow start \
  --type AgentPipelineWorkflow --task-queue agent-pipeline \
  --workflow-id demo-boat-1 \
  --input '"research-buddy"' \
  --input '"My boat is a Mastercraft Maristar 245."' \
  --input '"What boat do I have?"'

You should see output including (abridged — exact formatting varies by temporal version):

Running execution:
  WorkflowId  demo-boat-1
  Type        AgentPipelineWorkflow
  TaskQueue   agent-pipeline

The type AgentPipelineWorkflow and the queue agent-pipeline are fixed by the sample: they must match what the worker registered, or the work sits in a queue nobody is watching and the workflow simply never starts.

Remember the shared-namespace warning — if demo-boat-1 is already taken by someone else's run, pick a different id.

4. Watch it complete

kubectl exec -n cai-services "$POD" -- temporal workflow describe \
  --workflow-id demo-boat-1

You should see output including:

Status  COMPLETED

Still showing Running? That's usually the agent cold-starting after being idle — the first activity attempt times out at 30 seconds, waits a second, and tries again. Give it another moment and re-run the command. What you are watching is the retry policy doing precisely the job you'd otherwise be writing by hand.

5. Read the history

This is the payoff. The engine recorded every scheduled task, every attempt, every result:

kubectl exec -n cai-services "$POD" -- temporal workflow show \
  --workflow-id demo-boat-1

You should see two ActivityTaskCompleted events — both from the InvokeAgent activity — and a final result shaped like:

{"First":{"session_id":"...","output":"..."},"Second":{"session_id":"...","output":"..."}}

Two things to check in that result:

1. Second.session_id is the same as First.session_id. The workflow threaded the session through.
2. Second.output answers the boat question. The agent remembered the first message because it was in the same session — the workflow didn't repeat the boat details.

That history is durable and queryable after the fact. When someone asks "did the pipeline actually call the agent twice on Tuesday?", this is the answer, not a guess reconstructed from logs.

6. Watch a failure exhaust its retries

Now make it fail on purpose, by naming an agent that doesn't exist:

kubectl exec -n cai-services "$POD" -- temporal workflow start \
  --type AgentPipelineWorkflow --task-queue agent-pipeline \
  --workflow-id demo-missing-1 \
  --input '"no-such-agent"' \
  --input '"hello"' \
  --input '"hello again"'

Wait a few seconds — the attempts are one second, then two seconds apart — then look:

kubectl exec -n cai-services "$POD" -- temporal workflow describe \
  --workflow-id demo-missing-1

You should see:

Status  FAILED

And the history shows the attempts, not just the ending:

kubectl exec -n cai-services "$POD" -- temporal workflow show \
  --workflow-id demo-missing-1

You should see repeated ActivityTaskFailed events for InvokeAgent, stopping after the third — the MaximumAttempts: 3 ceiling from the retry policy. The second activity never ran, because the first never produced a session to hand it.

This is the behavior you want from a pipeline: bounded retries, then a clear, recorded failure. Not an infinite loop, and not a silent stop.

7. Clean up

Remove the worker. The completed and failed workflow histories stay in the engine — that's the point of them:

kubectl delete -f examples/workflows/agent-pipeline/deploy.yaml

You should see:

deployment.apps "agent-pipeline-worker" deleted

Delete the agent too if you're finished with it:

platformctl delete research-buddy

Writing your own workflow

There's no template generator yet. Copy examples/workflows/agent-pipeline and change four things:

1. The workflow and activity code. Keep activities small and independently retryable — one network call per activity is a good rule, because an activity is the unit that gets retried.
2. The task queue name. Give yours its own, or your worker will pick up the sample's work.
3. The worker Deployment. It must run in-cluster and point TEMPORAL_ADDR at temporal-frontend.cai-services.svc.cluster.local:7233. There is no external endpoint to connect a laptop to.
4. The retry policy, if 3 attempts and a 30-second timeout don't suit your steps.

Things your workflow code should honor, given the alpha constraints:

Constraint	What to do about it
Shared workflow namespace across all projects	Prefix every workflow ID with your project or team name
No authentication on the engine	Never pass secrets as workflow inputs — read them at activity time from Secrets Manager
Single-replica engine components	Expect pauses across restarts; make activities idempotent so a retry after a pause is harmless
Activities call agents over HTTP	An agent may be cold; give an activity enough StartToCloseTimeout to survive a cold start

Next steps

• Run a workflow — the same sample as a compact how-to you can keep open while working.
• Durable Execution overview — the mental model, and an honest comparison with AWS Step Functions, GCP Workflows, and Azure Durable Functions.
• Invoke an agent — the endpoint each activity calls, and its response shape.
• Sessions — why passing session_id to the second call makes the agent remember.
• Event-driven functions — the other way to run code in response to something happening.