Tool calling

Your tools never run on the platform. There is no hosted executor — a tool executes in your page or on your server, with your credentials, under your authorization rules. That is a deliberate boundary, not a gap we forgot to fill.

The reason is trust. A hosted tool runner would need your credentials, your authorization rules, and a say in when a side effect fires. Your backend already has all three and already knows which user is on the call. So execution stays where the trust already is, and you get two lanes to wire it in: client tools, where she calls a function you declared in your page mid-conversation, and server steering, where your backend runs the agent loop and hands her the result to say.

Client tools: functions she can call in your page

Opt in at mint with clientTools: true (wire capabilities: ["client_tools"]) — part of the call policy, so it is your server's decision like everything else — then declare the tools in the page after the room connects. The manifest is registered over RPC; nothing about a tool ever appears in the mint request, and the tool name is the record key.

// Server — the connect route opts the call in. Policy, like everything else.
session: async () => ({ instructions, clientTools: true }),

// Page — after room.connect(). The name she calls is the record key.
import { attachAvatarTools, type AvatarTool } from "realtime-avatar/tools";

const check_order: AvatarTool<{ order_id: string }> = {
  description: "Look up the status of an order by id",
  parameters: {
    type: "object",
    properties: { order_id: { type: "string" } },
    required: ["order_id"],
  },
  execute: async ({ order_id }) =>
    await fetch("/api/orders/" + order_id).then((r) => r.json()),
};

const { accepted, rejected } = await attachAvatarTools(room, { check_order });

A tool has 2.5 seconds. That is a conversational floor, not a tunable — past a couple of seconds of dead air the call stops feeling live. The platform abandons the call after the deadline and tells her it failed, and the abort is cooperative: every execute receives an AbortSignal, but a handler that ignores it still runs to completion and can commit a side effect — only its result is discarded. Make anything slow idempotent, or check signal.aborted before you write. A tool that calls a model does not fit: return an acknowledgement inside the deadline and deliver the real answer on screen when it lands. And read the { accepted, rejected } return — a rejected schema is a tool she simply does not have.

Server steering: the two primitives

Per-turn instructions is the tool-result channel. It steers wording, register, and content for one reply and then evaporates — it does not mutate the session prompt, so a tool result cannot leak into every later turn.

The loop

Run your agent where your data is: on your server. The call is the interface, not the brain.

// 1. You observe the user's turn. 2. Your agent decides a tool is needed
// and runs it. 3. You hand the RESULT back and let her say it.
const result = await tools.checkOrderStatus({ userId, orderId });

await session.sendTurn(userText, {
  instructions: [
    "Answer using ONLY these facts:",
    JSON.stringify(result),
    "One or two spoken sentences. Do not read the JSON aloud.",
  ].join("\n"),
});

import json
from fastapi import Depends

# The live turn is client-side, so Python's role is to DECIDE and to receive.
# Your client calls your endpoint; you run the tool and return the steer text.
@app.post("/api/turn")
def turn(req: TurnRequest, user=Depends(current_user)):
    result = check_order_status(user.id, req.order_id)
    return {
        "text": req.text,
        "steer": (
            "Answer using ONLY these facts: "
            f"{json.dumps(result)}. "
            "One or two spoken sentences. Do not read the JSON aloud."
        ),
    }

For a deterministic string — a confirmation number, a price, a required disclosure — do not route it through the model at all. Use the verbatim path so what she says is exactly what you wrote.

Latency

A tool call between the user finishing a sentence and the character starting to speak is dead air, and dead air on a video call is louder than on a chat. Two things help:

• Speak first, then resolve. Send a short acknowledging turn immediately ("let me look"), run the tool, then send the answer turn.
• Pre-fetch at mint. Anything you can know before the call starts belongs in the policy's context (wire initial_context), not in a tool call three seconds in.

Getting the conversation back

Your agent needs to know what was said — and what she did. Register a transcript webhook at mint time (wire transcript_webhook: an https URL and a 16–200 character secret; tag the session with client_metadata) and you get a signed POST after the call ends with the full two-sided transcript (segments) and the tool calls the model acted on (tool_calls: name, arguments, result or error, and duration). It is sent after capacity is released, so it never delays the next caller. Delivery is at-least-once: the worker waits 5 seconds for your answer, retries once after ~2 seconds on a 5xx or a timeout with the identical body and signature, and takes any status under 500 as delivered. A session with no committed turns and no tool calls sends nothing:

// Registered on your connect endpoint — server-side only.
session: async ({ avatarId }) => ({
  instructions,
  transcript: { url: "https://your.app/api/rta/transcript", secret: TRANSCRIPT_SECRET },
  // Echoed verbatim, so the keys are yours — except user_id, which
  // GET /usage/sessions?endUserId= filters on.
  metadata: { user_id: user.id, characterId: avatarId, mode: "video" },
})

import hashlib, hmac, json, time
from fastapi import BackgroundTasks, HTTPException, Request

# Mint half — in your connect route, next to avatar_id. Wire names, not the
# SDK's transcript / metadata.
body["transcript_webhook"] = {"url": "https://your.app/api/rta/transcript",  # https only
                              "secret": TRANSCRIPT_SECRET}                   # 16..200 chars
body["client_metadata"] = {"user_id": str(user.id), "characterId": character.avatar_id,
                           "mode": req.mode}                                 # up to 16 string pairs

# Receiver half.
MAX_SKEW_SECONDS = 300


def verify(body: bytes, signature: str, timestamp: str, secret: str) -> bool:
    if not timestamp.isdecimal() or abs(time.time() - int(timestamp)) > MAX_SKEW_SECONDS:
        return False                                  # missing, malformed, or stale: replay-bound
    signed = f"{timestamp}.{body.decode()}".encode()
    expected = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(f"v1={expected}", signature)


@app.post("/api/rta/transcript")
async def transcript(request: Request, tasks: BackgroundTasks):
    raw = await request.body()                        # RAW bytes, before any parsing
    if not verify(raw, request.headers.get("x-rta-signature", ""),
                  request.headers.get("x-rta-timestamp", ""), TRANSCRIPT_SECRET):
        raise HTTPException(401)
    payload = json.loads(raw)
    if already_saved(payload["session_id"]):          # at-least-once: dedupe on session_id
        return {"ok": True}
    user_id = (payload.get("client_metadata") or {}).get("user_id")   # None when the mint set none
    tasks.add_task(save_turns, user_id, payload["segments"])
    tasks.add_task(save_tool_calls, user_id, payload.get("tool_calls", []))   # absent when none ran
    return {"ok": True}                               # answer inside 5 s; save after

Registering it without the SDK. The TypeScript tab above turns the webhook on; the Python tab only verifies what arrives. If your server mints over raw HTTP, the field on the mint body is transcript_webhook — not transcript, which is the SDK's name for it — and the mint is strict, so the wrong key is a 422 rather than a dropped field and no webhook is ever sent. secret must be 16–200 characters: it is the HMAC key the handler below verifies against, so generate it rather than typing one.

The POST carries x-rta-signature: v1=<hex hmac> and x-rta-timestamp; the signed payload is "<timestamp>.<body>" under HMAC-SHA256 with your shared secret. Verify it before trusting anything, and reject stale timestamps. client_metadata comes back verbatim ({} when the mint sent none), so you can attribute the transcript without a session lookup, and session_id is the key to dedupe a retry on.

tool_calls is present only when the session ran at least one tool. Each entry is {name, call_id, arguments, ts, ok, result | error, duration_ms}; an entry with no ok field means the call produced nothing the model saw. Store it next to the segments — a reply grounded in a lookup is only auditable if the lookup itself is in the history. Arguments and results are truncated to 2,000 characters each and at most 200 calls are recorded (tool_calls_truncated: true marks an overflow); this is a history, not a replay. One known gap: a tool round belonging to a turn the user barged in over is not reported by the runtime, so its calls are absent even though they ran.

The platform stores no conversation text. If you do not register a webhook, nothing is buffered and nothing is sent — which is the right default for most apps, and the only acceptable one for some.

Bringing your own model

If your agent loop is complex enough that per-turn steering feels like fighting it, you can go further: mint with listen: false (wire stt_mode: "off"), keep speech recognition on your side, and drive every turn through sendTurn with tight instructions. At that point the platform is a rendering endpoint for a character you fully control, and every tool decision is yours. That is a supported way to use it.

You can also select the brain per session with llm (provider + model) when you want the hosted loop but a different model behind it.