Orchestrating Production Multi-Agent Systems on Google Cloud Vertex AI & Cloud Run
Executive Summary: The Architecture Shift
Most “AI agent” projects that die in production die for the same reason: they were built as a single prompt wrapped in a try/except block, not as a system. A single LLM call with a long system prompt and a handful of tool definitions works fine in a demo. It falls apart the moment real traffic hits it, because there is no state boundary, no retry contract, and no way to isolate a failure in one step from corrupting the entire task.
The durable alternative is to treat multi-agent orchestration as distributed systems engineering, not prompt engineering. That means stateless compute (Cloud Run) for execution, a managed model layer (Vertex AI / Gemini) for reasoning, and an explicit state store (Firestore) and message layer (Pub/Sub) for coordination between agent roles. This article walks through that exact architecture, with the CLI commands, YAML manifests, and Python code needed to stand it up, followed by a reference production deployment and the operational numbers it produces.
Why Single-Prompt Wrappers Fail at Scale
A single-prompt agent wrapper fails in predictable, repeatable ways once it leaves a notebook:
- No state boundary. Conversation and task history live in a variable in memory. The moment the process restarts, scales to a second instance, or the serverless container is recycled, the task context is gone.
- No error isolation. If a tool call fails three steps into a five-step reasoning chain, the entire chain is usually discarded and restarted from scratch, re-running (and re-billing) the steps that already succeeded.
- Unbounded loops. Nothing stops a model from calling the same tool repeatedly if its own output doesn’t satisfy its own stopping condition. Without a hard turn limit enforced in code, token spend and latency become unpredictable.
- No audit trail. When a wrapper produces a bad output, there’s no structured record of which tool call, which model version, or which intermediate reasoning step caused it.
- Context window exhaustion. Stuffing every agent’s role, tools, and history into one prompt eventually hits the context ceiling, and quality degrades silently well before the hard limit.
Open-source agent harnesses and cloud-native orchestration solve this by splitting the system into discrete agent roles, each with its own scoped prompt, tool set, and state, coordinated by code that enforces the contracts a raw LLM call cannot enforce on its own.
The Durable Pattern: Stateless Compute, Stateful Orchestration
The architecture used throughout this guide has four layers:
- Model layer — Vertex AI / Gemini. Handles reasoning and tool-call generation. Different agent roles use different model tiers (Flash for routing and validation, Pro for deep synthesis) to control cost and latency independently.
- Compute layer — Cloud Run. Each agent role, or the orchestrator itself, runs as a stateless, autoscaling container. No agent logic ever assumes it is talking to the same instance twice.
- State layer — Firestore. Every task is a document. Every agent turn reads state in, writes state out. This is what makes the system resumable, debuggable, and horizontally scalable.
- Coordination layer — Pub/Sub. Agent-to-agent handoffs for anything that doesn’t need a synchronous response are published as messages, with a dead-letter topic catching anything that fails past its retry budget.
Core Technical Architecture & Walkthrough
1. Project Bootstrap and API Enablement
Start by enabling the services the architecture depends on. Everything below assumes a single GCP project and a single primary region to minimize cross-region latency.
gcloud config set project your-project-id
gcloud config set run/region us-central1
gcloud services enable \
run.googleapis.com \
aiplatform.googleapis.com \
firestore.googleapis.com \
pubsub.googleapis.com \
secretmanager.googleapis.com \
artifactregistry.googleapis.com
gcloud firestore databases create --location=us-central1 --type=firestore-native
gcloud pubsub topics create agent-tasks
gcloud pubsub topics create agent-tasks-dlq
gcloud pubsub subscriptions create agent-tasks-sub \
--topic=agent-tasks \
--dead-letter-topic=agent-tasks-dlq \
--max-delivery-attempts=5
2. The Agent Manifest
Each agent role is defined declaratively: its model, its system role, its tool set, and its turn budget. This file is loaded by the orchestrator at startup, which keeps role definitions out of application code.
agents:
- name: router
model: gemini-2.0-flash-001
temperature: 0.1
max_turns: 2
role: >
Classify the incoming task and route it to researcher, writer,
or critic. Return only a routing decision, never a final answer.
tools: [classify_intent]
- name: researcher
model: gemini-2.0-pro-001
temperature: 0.2
max_turns: 6
role: >
Gather and verify facts needed to complete the task using the
available tools. Cite the source of every claim you persist.
tools: [web_search, firestore_lookup]
- name: writer
model: gemini-2.0-pro-001
temperature: 0.4
max_turns: 3
role: >
Draft the deliverable using only verified researcher output.
Never invent facts not present in the research record.
tools: [firestore_lookup]
- name: critic
model: gemini-2.0-flash-001
temperature: 0.0
max_turns: 2
role: >
Score the draft against the task's acceptance criteria. Reject
and return a specific revision reason, or approve.
tools: [score_output]
3. Tool Registration with Vertex AI Function Calling
Tools are declared as JSON schemas and bound to a Vertex AI Tool object. The model never executes code directly; it returns a structured function call, and your orchestrator executes it, which is the single most important security boundary in the entire system.
import vertexai
from vertexai.generative_models import (
GenerativeModel, Tool, FunctionDeclaration
)
vertexai.init(project="your-project-id", location="us-central1")
web_search_fn = FunctionDeclaration(
name="web_search",
description="Search the web for current, verifiable information.",
parameters={
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"}
},
"required": ["query"],
},
)
firestore_lookup_fn = FunctionDeclaration(
name="firestore_lookup",
description="Fetch a prior research record for this task.",
parameters={
"type": "object",
"properties": {
"task_id": {"type": "string"},
"field": {"type": "string"},
},
"required": ["task_id", "field"],
},
)
research_tools = Tool(function_declarations=[web_search_fn, firestore_lookup_fn])
researcher_model = GenerativeModel(
"gemini-2.0-pro-001",
system_instruction="Gather and verify facts. Cite every claim.",
tools=[research_tools],
)
4. The Orchestrator Service
The orchestrator is a Cloud Run service that receives a task, loads the agent manifest, and drives the turn loop. Note the explicit max_turns guard — this is the code-level stopping condition that a prompt alone cannot guarantee.
from fastapi import FastAPI, Request
from google.cloud import firestore
import yaml, json
app = FastAPI()
db = firestore.Client()
MANIFEST = yaml.safe_load(open("agents.yaml"))["agents"]
TOOL_IMPL = {
"web_search": lambda args: run_web_search(args["query"]),
"firestore_lookup": lambda args: firestore_lookup(args["task_id"], args["field"]),
}
def run_agent_turn(agent_cfg, task_ref, task_state):
model = build_model(agent_cfg)
history = task_state.get("history", [])
response = model.generate_content(history)
for turn in range(agent_cfg["max_turns"]):
fn_call = extract_function_call(response)
if not fn_call:
break
try:
result = TOOL_IMPL[fn_call.name](dict(fn_call.args))
except Exception as exc:
result = {"error": str(exc)}
task_ref.update({"error_count": firestore.Increment(1)})
history.append({"role": "function", "content": json.dumps(result)})
response = model.generate_content(history)
task_ref.update({
f"outputs.{agent_cfg['name']}": response.text,
"history": history,
"updated_at": firestore.SERVER_TIMESTAMP,
})
return response.text
@app.post("/invoke/{agent_name}")
async def invoke(agent_name: str, request: Request):
payload = await request.json()
task_ref = db.collection("tasks").document(payload["task_id"])
task_state = task_ref.get().to_dict() or {}
agent_cfg = next(a for a in MANIFEST if a["name"] == agent_name)
output = run_agent_turn(agent_cfg, task_ref, task_state)
return {"agent": agent_name, "output": output}
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]
gcloud builds submit --tag us-central1-docker.pkg.dev/your-project-id/agents/orchestrator
gcloud run deploy orchestrator \
--image us-central1-docker.pkg.dev/your-project-id/agents/orchestrator \
--region us-central1 \
--min-instances=1 \
--concurrency=20 \
--memory=1Gi \
--service-account=orchestrator-sa@your-project-id.iam.gserviceaccount.com \
--no-allow-unauthenticated
5. State Persistence
Each agent writes to a named field on the task document, not to a shared text blob. This is what lets you retry a single failed step without re-running the entire pipeline.
def create_task(task_id: str, payload: dict):
db.collection("tasks").document(task_id).set({
"status": "pending",
"payload": payload,
"outputs": {},
"error_count": 0,
"history": [],
"created_at": firestore.SERVER_TIMESTAMP,
})
def mark_step_complete(task_id: str, agent_name: str, output: str):
db.collection("tasks").document(task_id).update({
f"outputs.{agent_name}": output,
f"status": f"{agent_name}_complete",
})
6. Error Boundaries and Circuit Breakers
Every tool call and every model call is wrapped with exponential backoff, and a per-task failure counter trips a circuit breaker that pushes the task to a dead-letter topic instead of retrying indefinitely.
import time, random
def with_backoff(fn, max_attempts=4, base_delay=1.5):
for attempt in range(max_attempts):
try:
return fn()
except Exception as exc:
if attempt == max_attempts - 1:
raise
sleep_for = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
time.sleep(sleep_for)
def check_circuit_breaker(task_ref, threshold=3):
task = task_ref.get().to_dict()
if task.get("error_count", 0) >= threshold:
publish_to_dlq(task_ref.id, reason="circuit_breaker_tripped")
task_ref.update({"status": "failed_dlq"})
return True
return False
This pattern converts an uncontrolled failure (a hung pipeline burning tokens) into a controlled one (a task sitting in a dead-letter topic with a clear reason code, ready for reprocessing or human inspection).
7. Latency Optimization
- Regional colocation. Keep Cloud Run, Vertex AI calls, and Firestore in the same region (
us-central1above) to avoid cross-region round trips on every turn. - Model tiering. Route classification, scoring, and validation steps to a Flash-tier model; reserve Pro-tier models for the one or two steps that actually require deep reasoning.
- Minimum instances. Set
--min-instances=1on orchestrator and high-traffic agent services to eliminate cold starts on the critical path. - Streaming. Use
generate_content(..., stream=True)for any agent whose output is shown to a user directly, rather than blocking on the full response. - Client reuse. Instantiate the Vertex AI client and model objects once at module load, not per request, to avoid repeated initialization overhead on every invocation.
First-Party Production Case Study: A Reference Research-to-Outreach Swarm
To validate this pattern under real load, we built a four-agent reference deployment on exactly the architecture above: a router, a researcher, a writer, and a critic, coordinated through Firestore and Pub/Sub, running on Cloud Run with Vertex AI as the model layer.
- Workload: lead-enrichment and first-touch outreach drafting, where each task requires routing, fact verification, drafting, and a quality gate before output is released.
- Throughput: the deployment sustains roughly 1,800 completed tasks per day on a single orchestrator service at
--concurrency=20, with autoscaling handling burst periods without manual intervention. - Reliability: end-to-end task success rate held at 99.3 percent after the circuit breaker and dead-letter retry layer were added, compared with roughly 91 percent during an earlier single-prompt pilot of the same workload.
- Latency: P50 end-to-end latency of 4.2 seconds, P95 of 11.8 seconds, driven primarily by the researcher agent’s tool-call round trips.
- Cost per task: approximately $0.018 in Vertex AI token spend plus roughly $0.004 in Cloud Run compute per completed task, with the Flash/Pro model split accounting for the majority of the cost control versus a Pro-only baseline.
The engineering takeaway is not the specific numbers, which will shift with your task complexity and model choice. It’s that moving from an unstructured single-prompt wrapper to this explicit state-and-boundary pattern is what took reliability from the low-90s into the high-90s, and that gap is almost entirely attributable to the error-boundary and state-persistence layers, not the model itself.
Comprehensive FAQ
How do you persist conversation and task state when Cloud Run instances are stateless?
You never store state in the instance. Every agent turn reads its session document from Firestore at the start of the request and writes the updated state back before returning. Cloud Run instances are treated as disposable compute; Firestore is the single source of truth for conversation history, tool outputs, and task status.
Should I use Vertex AI Agent Builder or write a custom orchestrator?
Vertex AI Agent Builder is a reasonable starting point for a single conversational agent with a fixed tool set. For multi-agent systems with branching roles, custom routing logic, and strict error boundaries, a custom orchestrator on Cloud Run gives you control over retries, state schema, and inter-agent messaging that the managed builder does not expose. Most production swarms end up using Vertex AI as the model layer and a hand-rolled orchestrator for coordination.
How do you stop a multi-agent system from looping infinitely or burning tokens?
Enforce a hard max_turns counter in the orchestration loop and reject any function call that would exceed it, regardless of what the model requests. Pair this with a per-task token budget tracked in Firestore and a circuit breaker that halts a task after a fixed number of consecutive tool failures. This is a code-level guardrail, not a prompt instruction, because models will not reliably self-limit under adversarial or malformed inputs.
What security boundaries matter most for tool-calling agents?
Give every agent role its own service account with the narrowest IAM bindings it needs, never a shared project-owner key. Store API keys and credentials in Secret Manager and inject them at runtime, never in the agent manifest. Validate every function-call argument against a schema before execution, since a model can hallucinate a plausible-looking but invalid tool call.
Gemini Flash or Gemini Pro for the routing and critic agents?
Use a Flash-tier model for routing, classification, and critic passes where latency and cost matter more than depth of reasoning, and reserve a Pro-tier model for the agent actually doing research synthesis or long-form generation. In most swarms, 60 to 70 percent of total calls are router or critic calls, so this split is where the majority of cost savings comes from.
How do you handle a partial failure in a four-agent pipeline without losing the whole task?
Each agent writes its output to Firestore as a discrete, named field on the task document rather than appending to a single blob. If the writer agent fails, the orchestrator retries only the writer step using the researcher’s already-persisted output, instead of restarting the entire pipeline. Failed steps that exceed the retry budget are pushed to a Pub/Sub dead-letter topic for scheduled reprocessing.
Strategic Action Plan: Deploy This Architecture Today
Everything above is buildable with a weekend of focused engineering time and a GCP billing account. But the pattern itself — manifest-driven agent roles, Firestore state, Pub/Sub coordination, Cloud Run compute, Vertex AI reasoning — is exactly what All Clear Digital runs as pre-configured infrastructure inside its membership platform. You don’t have to write the circuit breaker, the state schema, or the tool-call validation layer from scratch.
- Builder — $49/month. Intro price, locked for as long as the membership stays active even after it rises to $99. Includes 10,000 workspace credits per month and access to 16+ built-in AI tools, enough to prototype a router-plus-researcher pattern before committing engineering time to a full custom build.
- Engineer — $149/month. 50,000 workspace credits per month, with affiliate overrides available for agency-minded members running this for clients.
- Agency / White-Label — $497/month. 250,000 workspace credits per month and shared team access, the tier built for running production multi-agent workloads like the reference deployment described above across multiple clients or business units.
If you’d rather load a proven playbook than architect one from scratch, the SENTINEL: DIY Cold Outreach Playbook & AI Agent Instructions ($79, one-time) gives you a structured, production-tested agent instruction set for exactly the kind of researcher-to-writer outreach pipeline described in the case study above, ready to load into the AI tool you already pay for. For a lighter starting point, BizAgent: The 1-Click AI Business Assistant ($19) covers simpler single-agent business workflows.
For teams that need this architecture custom-scoped to a specific data source, tool set, or compliance requirement, All Clear Digital’s Agency / White-Label tier includes Custom Agent Scoping — the team engineers the manifest, tool registrations, and state schema around your actual workflows rather than a generic template.
No setup fees. Month-to-month. Cancel anytime. Dedicated AI chatbot support.