Creating an avatar

Between sentences your character is not speaking — and that is most of a conversation. A still portrait reads as a dead connection. You do not have to fix that yourself any more: creation takes one image and generates the motion for you.

One image in, a moving character out

Upload a portrait, create the avatar, and the platform renders — in the background — everything a live call needs:

• The looping idle video: a ~10s closed arc whose first and last frame are both the portrait, so it loops seamlessly by construction. This becomes the avatar's source video — the state she rests in.
• A multi-clip motion library: a motion director looks at the portrait and the persona and designs short closed-arc clips — a signature idle variant, a listening state, a gesture — each ending exactly where it began, so the realtime worker can swap between them at invisible seams.

The create call returns immediately with status: "preprocessing" (the wire also reports the loop's own idleVideoStatus); poll getAvatar(id) (GET /avatars/{id} on the wire) until ready, and read the library with GET /avatars/{id}/clips. Render calls inherit the generated library automatically; a mint that sends an explicit empty clip_library: [] opts that call out of clips entirely. A motion prompt at creation art-directs the loop (wire motionPrompt, or the Avatars page's Resting motion direction); a clip that fails to render fails alone — the avatar still ships with the loop and the clips that made it.

Creation is image-only: one portrait is the entire input, and the platform generates the initial videos. Afterwards, library clips can also use pose-compatible uploaded video assets. Below are the session-time behavior, the full-library declaration, and the generation tools you can use to art-direct motion.

Two modes

session: async ({ avatarId }) => ({
  video: { mode: "generative" },   // that is the entire switch
})

body = {
    "avatar_id": avatar_id,
    "mode": "avatar",
    "render_backend": "generative",   # that is the entire switch
}

Generative ignores clips by construction — the type makes passing them impossible rather than silently dropping them. Everything below is looping.

A live call needs a video source — which every avatar created from an image now grows on its own. Before the generated loop attaches, a mint is refused with a clear error, never a black or broken call; once it attaches, calls can connect while the realtime cache still warms (the first call starts slower). Poll status to ready for the fully-warm contract; a failed status carries the reason in error.

Looping: the resting loop

You do not need a session override. The avatar's source video is the generated resting loop, and calls inherit its clip library and behavior. Resting in that loop needs no declaration at all:

session: async ({ avatarId }) => ({
  // no video config: the avatar rests in its generated idle loop
})

body = {
    "avatar_id": avatar_id,
    "mode": "avatar",
    # no video fields: the avatar rests in its generated idle loop
}

Looping: shared clips, event-driven behavior

A clip is media; behavior decides when to play it. The same clip can serve the idle variations, the on.userSpeechStarted.clips list, and any action under actions at once. There are no per-clip roles. The platform library and its behavior ride calls automatically.

userSpeechStarted plays a listening reaction once when the user starts speaking; it is not a state held for their whole utterance. actions is the requestable catalogue: the language model reads each action's description and asks for one when it fits its reply — declaring an action never plays it by itself. Write that description for the language model, not as a generation prompt: the renderer reads source.motionPrompt instead. Speaker lip-sync stays on the current playback cursor during full-duplex speech; speaking does not reset the body to a separate talking loop.

GET separates media and job rows in data from playback rules in behavior. Rows have no row-level role, whenHint, or weight; the one weight lives on the idle group. Inspect both together:

const library = await rta.listClips(avatarId);
const { data, revision, anchor, defaultSourceAssetId, behavior } = library;
// data: media/job rows; url is the serving take, status is the current job.
// behavior.idle: { clips: ["idle_soft", "nod"], weight: 2 }
// behavior.on?.userSpeechStarted: { clips: ["nod"] }
// behavior.actions: {
//   agree: { description: "When agreeing with the user", clips: ["nod"] }
// }

response = client.get(f"/avatars/{avatar_id}/clips")
response.raise_for_status()
library = response.json()
revision = library["revision"]   # 0 = creation-owned; bumps on every accepted PUT
clips = library["data"]         # media/job rows, not behavior declarations
anchor = library["anchor"]
default_source_asset_id = library["defaultSourceAssetId"]
behavior = library["behavior"]
# behavior["idle"] = {"clips": ["idle_soft", "nod"], "weight": 2}
# behavior["on"]["userSpeechStarted"] = {"clips": ["nod"]}
# behavior["actions"]["agree"] = {
#     "description": "When agreeing with the user", "clips": ["nod"]}
# idle, on and actions are optional; inspect their presence before indexing.

A mint that sends an explicit empty clip_library: [] opts that call out of clips entirely — she stays in the resting loop. For persistent behavior changes, declare the avatar library rather than building a second client-side state machine.

One weighted idle pool

The stored source is addressed as primary and participates in the same weighted idle pool as the authored clips. Use idle.weights to weight any member, including the source; zero disables it. Every member avoids immediately repeating when another is available, while a sole available member loops completely. The id primary stays reserved for source addressing.

{
  "idle": { "clips": ["sway", "stretch"], "weights": { "primary": 1, "sway": 2, "stretch": 1 } }
}

Each member weight is finite, from 0–100, and defaults to 1. At least one member must have positive weight. Unavailable members are removed before one weighted draw; the remaining weights are renormalized. These are selection probabilities, not screen-time proportions. Use weights or the legacy aggregateweight, never both. Legacy idle.weight defaults to 1 and is divided equally among listed clips, with source weight 1. No media is re-rendered by this migration. If a disabled source is the only registered material, session admission returns a retryable clip_library_not_ready.

Every clip is a sentence, and you can change it

The starter library is a starting point, not a fixed set. Declare the library you want and the platform reconciles: a clip whose description is unchanged is kept and keeps serving, a new or revised one is queued to render, and one you leave out is retired. The 202 is acceptance, not readiness. revision is the top-level field of the GET envelope beside data — 0 while the creation pipeline owns the library; each accepted declare increments it, and the 202 body carries the new value.

This is a full desired set: removing a record retires that clip, and omitted behavior is removed too. expectedRevision is required. Read the library first, keep the sources you still want, and send the revision you read; a stale writer receives 409 revision_conflict. For older generated rows whose editable motion direction is null, supply a new direction before declaring them as generated sources.

const { revision } = await rta.listClips(avatarId);
const { plan } = await rta.setClipLibrary(avatarId, {
  expectedRevision: revision,          // required compare-and-set
  clips: {
    idle_soft: { source: { motionPrompt: "breathing gently, a slow blink" } },
    nod: { source: { motionPrompt: "a small attentive nod, returns to rest", durationSeconds: 5 } },
    wave: { source: { motionPrompt: "raises a hand and waves warmly, then lowers it" } },
  },
  idle: { clips: ["idle_soft", "nod"], weight: 2 },
  on: {
    userSpeechStarted: { clips: ["nod"] },
  },
  actions: {
    agree: { description: "When agreeing with the user", clips: ["nod"] },
    greet: { description: "When greeting the user", clips: ["wave"] },
  },
});
// plan → { kept: [...], queued: [...], retired: [...] }

await rta.waitForClips(avatarId);      // settles when nothing is still RENDERING

response = client.get(f"/avatars/{avatar_id}/clips")
response.raise_for_status()
revision = response.json()["revision"]
r = client.put(f"/avatars/{avatar_id}/clips", json={
    "expectedRevision": revision,   # required compare-and-set
    "clips": {
        "idle_soft": {"source": {"motionPrompt": "breathing gently, a slow blink"}},
        "nod": {"source": {"motionPrompt": "a small attentive nod, returns to rest", "durationSeconds": 5}},
        "wave": {"source": {"motionPrompt": "raises a hand and waves warmly, then lowers it"}},
    },
    "idle": {"clips": ["idle_soft", "nod"], "weight": 2},
    "on": {
        "userSpeechStarted": {"clips": ["nod"]},
    },
    "actions": {
        "agree": {"description": "When agreeing with the user", "clips": ["nod"]},
        "greet": {"description": "When greeting the user", "clips": ["wave"]},
    },
})
# httpx does not raise on 4xx: 403 clip_library_not_enabled, 409 revision_conflict
# (body carries the current revision), 422 clip_declaration_rejected
r.raise_for_status()
plan = r.json()["plan"]

Clip declarations are enabled by default for active workspaces. An operator-disabled workspace receives 403 clip_library_not_enabled; its existing clips keep serving. The other refusals — 409 revision_conflict, 422 clip_declaration_rejected — are in Refusals worth telling apart.

Each record contains only source: either { assetId } or { motionPrompt, durationSeconds? }. Generated source.durationSeconds is an integer from 4–8 seconds, default 5; it does not belong beside source. Changing source content queues media work; changing selection membership, the idle weight, or action descriptions is metadata-only and does not re-render unchanged media. An updated clip keeps its previous serving take while the new job runs, and after a failed replacement.

Listening and action lists draw uniformly. Idle uses the weighted pool above: each idle.weights value is finite in 0–100 and defaults to 1. Legacyidle.weight remains supported without weights. Each list is nonempty with unique references to records in clips; the source is addressed separately as primary in the weight map. Selection uses only eligible serving material.

Ten idle variations are valid; there is no role-count or total-clip authoring cap. Request bytes remain bounded by transport limits, and worker admission is independent: a valid declaration is not a promise of unlimited resident media or GPU capacity. Clip and action ids use 1–64 letters, digits, underscores, or hyphens; reserved keys __proto__, constructor, and prototype are refused. Motion prompts are nonempty after trimming and at most 1,200 characters; action descriptions are nonempty after trimming and at most 512 characters.

You can also upload a clip — source: { assetId } instead of a prompt — but it is pose-validated before it can serve: its first and last frames have to land on the avatar's rest pose, or that clip settles failed with the verdict in poseCheck and the rest of the library is untouched. That validation is the whole reason an upload is allowed at all; without it a supplied video makes every switch visible.

An action has its own id: greet is what the model asks for, and wave is only the media it plays — so several actions can share one clip, and one action can offer several for variety.

The resting loop is not a clip, and it is also a sentence

The avatar's source is the resting loop, and declaring idle variations does not replace it — resting is the other half of every idle draw, always available, and it is the one thing that may play twice running. So there is nothing to preserve and nothing to re-declare: the id primary is reserved, and a declaration that uses it is refused.

Do not declare the source asset as a variation. That adds a second copy of it: the avatar rests on it AND draws it as a variation, so it plays roughly twice as often as you intended and can follow itself back-to-back through the variation path — exactly the stutter the no-immediate-repeat rule exists to prevent. defaultSourceAssetId is on the listClips envelope to tell you which asset the loop is, not to be declared.

Direct the source loop at creation with motionPrompt, and re-direct it afterwards — that is how the resting loop changes, not through the clip library:

const { servingUrl } = await rta.setLoop(avatarId, {
  motionPrompt: "tilts her head, a small amused smile, settles back to centre",
});
// servingUrl = the loop she is playing RIGHT NOW, for the whole render

await rta.waitForLoop(avatarId);   // throws if the render gave up

accepted = client.put(f"/avatars/{avatar_id}/loop", json={
    "motionPrompt": "tilts her head, a small amused smile, settles back to centre",
}).json()
# accepted["servingUrl"] is the PREVIOUS loop (None if she had none yet) — she plays it for the whole render.
# Poll GET /avatars/{id} until idleVideoStatus is "ready" (or "failed").

She stays ready and keeps serving the previous loop for the entire render, then the swap publishes in one step. Your clip library is untouched — clips render against the portrait, not against the loop, so a re-direct re-queues nothing and does not move the library's revision. Describe a closed arc: it has to end where it began, or the loop snaps every time it wraps.

And keep it small. Because a separate model animates her mouth during speech, every loop description is held to a nearly-still envelope — blinks, a soft closed-lip smile, a shift in weight, a few degrees of head drift. A laugh, a head turn, or an open mouth comes back inside that envelope anyway: a new render, a new charge, and a loop that looks like the one you already had. It is the only outcome here that carries no error code, which is exactly what makes it worth knowing before you write the sentence.

What makes a switch invisible

Every clip the director renders starts and ends on the same rest pose — the portrait itself. That shared frame is what turns a switch into a splice instead of a jump, and it is why each clip is generated independently from the one anchor image rather than chained off the previous clip's last frame, which would compound drift with every hop. Clips stay short (~5s closed arcs): the lever for natural idle is how many states × how long she dwells, not clip length.

Editing the loop

Preview — not yet enabled on the public fleet. The field is accepted and forwarded today, and currently resolves to the unedited loop. Build against it if you want to be ready; do not ship a feature to your users that depends on it yet.

A state map switches between clips. Editing rewrites the clip itself: the generated loop is re-rendered under a sentence you write, and the result is lip-synced by the same path as an unedited one. One loop becomes every season, every set, every outfit — without ever generating a new one.

session: async ({ avatarId }) => ({
  video: {
    edits: { instruction: "a snowy cabin at night, warm lamplight" },
  },
})

body = {
    "avatar_id": avatar_id,
    "mode": "avatar",
    "support_edits": {"instruction": "a snowy cabin at night, warm lamplight"},
}

Editing runs upstream of the lips, on the closed-mouth plate, so it costs nothing at the start of a call — she is on screen and talking from the first word, she simply has not changed clothes yet. The first lap through the loop is live and expensive; every lap after replays what that one banked, so a fixed instruction converges on the cost of plain looping.

It needs the avatar's video source — the generated loop — so it is refused at the mint while the loop is still rendering, rather than falling back. It also cannot be combined with generative — a synthesized body has no clip to open — and the type makes that pairing unrepresentable rather than a runtime error. A voice call is refused for the same reason: no video track, nothing to rewrite.

Editing is per call, and off unless you ask. Nothing stored on the character turns it on, so a call that omits edits is byte-identical to one made before this option existed — which matters, because the first edited lap holds a scarce GPU editor and can wait minutes on a cold one. If you want every call for a character edited, say so in your session policy; that is the same place you already decide the persona and the ceiling.

Letting the look follow the conversation

Experimental — "runtime edit". The shape of live may still change, and the re-dress does not reach every capacity tier yet, so a call may run to the end on its opening set. See Experimental features for what that word promises. The trust boundary below is not experimental — it is enforced on every mint.

Add live and the set stops holding still: the look follows the conversation, within the bounds you set. Those rules are yours, not your user's — they are the app's policy about what a conversation is allowed to do to the picture, which is why they are server-owned like instructions. A browser that could set them could redress your character into anything.

edits: {
  instruction: "her apartment at golden hour",
  live: {
    rules: "only change the room and the light, never her face or clothes; " +
           "only when they ask to go somewhere",
    cooldownSeconds: 60,
  },
}

body["support_edits"] = {
    "instruction": "her apartment at golden hour",
    "live_edit": {
        "rules": ("only change the room and the light, never her face or clothes; "
                  "only when they ask to go somewhere"),
        "cooldown_seconds": 60,
    },
}

Write the rules narrowly. Every change of direction buys a fresh lap on a scarce editor and a visible style pop — the editor has no mid-stream prompt update, so a look changes at a seam rather than easing. cooldownSeconds is the floor that stops an eager director spending a GPU per user turn; it is a bound, not a preference, and the default is deliberately conservative.

Generation pricing

The generated library itself is currently included with creation — the loop's generation is metered per-model exactly like a resting-loop update on the Avatars page, and the extra library clips are not billed separately while pricing settles. The rates below apply when you drive the generators directly:

Generation is billed at the model vendor's own list price. We add no margin — what you pay is what MiniMax and ByteDance charge, so the table below moves when their rate cards move. Prices are per generated clip and charged on success; failed generations are not charged.

Prices are for a 10-second clip at 24fps: H3 at 768P and the Seedance tiers at 720p. The output follows the prepared source image; the model name does not select a landscape frame. New landscape uploads preserve their complete composition. The resting-loop editor changes motion while retaining the avatar's source framing; it has no model or aspect-ratio selector. An older avatar whose loop was already cropped needs its source framing repaired before regeneration. The Seedance rates come from ByteDance's published Seedance rate card ($0.15/sec at 720p, $0.12 Fast, $0.08 Mini). Credits bill at $1 ≈ 720 credit-seconds, so seedance-2.0 is 1,080 credits and the default minimax-h3 is 288 (a 10-second clip at $0.04/sec).

1080p costs materially more on every model — Seedance 2.0 is $0.37/sec there, roughly 2.5× the 720p rate. For a clip that spends its life as a background loop behind a speaking character, 720p is the right default.

Uploading the portrait

The one asset you bring is the image. Upload the bytes, or hand us a URL and we fetch it server-side — the second is what you want for anything large (the origin must answer with Content-Length; the same 8 MB image cap as an upload applies):

// bytes you already hold — createAvatar takes the asset id from here
const asset = await rta.uploadAsset(file, { kind: "image" });

// or a URL we fetch for you (no multipart through your runtime)
const asset = await rta.createRemoteAsset({
  kind: "image",
  remoteUrl: "https://your-bucket.example.com/rin/portrait.png",
});

asset.url; // https://realtimeavatar.ai/api/assets/… — public, ready to use

# client: the httpx.Client from the quickstart (base_url https://realtimeavatar.ai/api/v1, bearer API key)

# bytes you already hold
with open("portrait.png", "rb") as fh:
    asset = client.post("/assets", files={"file": fh}, data={"kind": "image"}).json()
# raw bytes without a filename: files={"file": ("portrait.png", data, "image/png")} —
# the server keys on the part's Content-Type (jpeg/png/webp)

# or a URL we fetch for you
asset = client.post("/assets/remote", json={
    "kind": "image",
    "remoteUrl": "https://your-bucket.example.com/rin/portrait.png",
}).json()

asset["publicUrl"]  # https://realtimeavatar.ai/api/assets/… — public, ready to use
# (the SDK surfaces this same field as asset.url; on the wire it is publicUrl)

The returned URL is publicly readable — no auth, no signing. Treat these URLs as unguessable, not private: the key is random and nothing indexes it, but anyone holding the link can fetch it. Do not upload anything whose exposure would matter.

Uploading costs nothing — you are charged for generation, not storage. Accepted kinds are image (portraits), audio (voice material), and video for an uploaded motion clip declared with source: { assetId }. Video upload does not make the asset an Avatar source: new Avatars are still portrait-only, and an uploaded action clip must pass the rest-pose check before it can serve.