Build an AI Customer Support Avatar That Knows When to Hand Off

Build an AI customer support avatar with human handoff by keeping conversation separate from business authority. Realtime Avatar handles the live voice or video interaction; your application authenticates the customer, retrieves permitted order facts, and creates support tickets. The avatar should announce a handoff only when your backend confirms what actually happened. A request to contact support is not a created ticket, and a created ticket is not a connected human.

Give each layer one job

Keep RTA’s managed conversation, but make account-specific operations application-owned. Your server sets session instructions and enables client tools; the client registers its tool manifest after connecting. Read the registration’s accepted and rejected results so a missing lookup tool cannot silently become an invitation to guess.

• The conversation layer recognizes a support request and asks clarifying questions. Instructions prohibit invented order facts, refund approvals, and unconfirmed transfer announcements.
• The application backend derives identity from the authenticated session, checks order ownership and operation permissions, and returns only the necessary facts. A spoken order number does not establish ownership.
• Your support integration creates tickets, checks the human queue, and reports assignment or connection. This is a pattern you build, not an automatic RTA CRM integration or contact-center routing feature.

Keep RTA call status separate from support status. The SDK’s waiting state describes avatar capacity, not a queue of human agents.

• Realtime Avatar tool execution and registration
• Session policy and call lifecycle

Walk through an order-status and refund request

Consider a fictional customer asking, “Where is order 4821? If it is late, refund it.” The lookup goes through your backend, scoped to the signed-in account. Suppose the authorized result says delayed, with no confirmed delivery date. The avatar can explain that fact; it cannot invent an arrival date or authorize a refund.

If policy requires human review, preview a redacted summary: “Order ending 4821 is delayed. Customer requests refund review. No refund has been issued.” Ask, “May I send this summary to support?” Let the customer edit or decline it. Keep full identifiers in access-controlled application records rather than spoken content; exclude addresses, payment details, and unrelated conversation.

After consent, persist the approved summary and create the support request idempotently. If your helpdesk returns ticket SUP-1042 and reports the queue closed, say, “Your request is saved as SUP-1042. Live support is unavailable now; follow its status here.” Show the receipt and authenticated status link. Quote a response estimate only when your support system supplies one.

Bind consent to the exact summary version, account, and order. An edited summary needs renewed approval, not reuse of a previous yes. Return allowlisted facts from tools rather than raw helpdesk records; free-text notes are data, not instructions. Test that a customer cannot swap an order ID or summary reference to disclose another account’s information.

An open queue permits offering a live transfer, not announcing a connection. Only a confirmed agent-joined event permits “A support agent has joined.” Drive both the screen and permitted spoken updates from these backend facts.

Respect the 2.5-second tool deadline

RTA tools have 2.5 seconds to return. Cancellation is cooperative: a handler can keep running and commit a side effect after its result is discarded. A timeout therefore proves neither failure nor cancellation. Slow ticket creation needs a durable backend job, not a browser promise left running.

Acknowledge only after that job is persisted, within the deadline. Return its real identifier and status location, clearly labeled pending. RFC 9110’s 202 Accepted semantics make the same distinction: processing has been accepted, not completed, and might never complete. An acknowledgement is not evidence of a ticket or refund.

This application pseudocode illustrates your backend boundary. It is not an RTA SDK call or wire payload.

// APPLICATION PSEUDOCODE — implement with your own services.
actor = requireAuthenticatedSession(request)
order = orders.requireOwned(actor.id, request.orderId)
permissions.require(actor, 'request_support', order)
approved = consent.requireCurrent(actor.id, request.summaryVersion)
key = handoffs.keyFor(actor.id, order.id, approved.id)
job = handoffs.enqueueOnce(key, approved.redactedSummary)
return accepted(job.id, statusUrlFor(job.id))
// A persisted job is not yet a helpdesk ticket.

The worker rechecks authorization, creates or reconciles the ticket, and persists its returned ID. Your UI follows that record beyond the call. Ending the RTA session does not cancel application jobs; cancellation needs its own authorized operation. For an exact closing receipt, the documented sayAndEnd path speaks verbatim and ends the avatar call, not the support case.

• RFC 9110, section 15.3.3: 202 Accepted

A failure matrix to test before launch

Exercise these branches against your integration. Each needs a backend assertion and a check of what the customer sees and hears.

• Authentication or ownership fails. Return no order details or existence confirmation. Offer sign-in or a general support channel, never account-specific speculation.
• The lookup exceeds its deadline. Keep the answer unknown. Say “Still checking” only if durable work really continues; otherwise explain that the lookup failed.
• Ticket creation times out after submission. Reconcile using the same idempotency identity before retrying. Show “Confirmation pending,” not an invented ticket number or a duplicate request.
• The human queue is closed or unreadable. Preserve any confirmed ticket and show its status link. Say unavailable or unknown respectively; do not promise an agent or invent a wait estimate.
• The customer declines summary sharing. Do not submit it. Offer a support entry point where they can explain the issue themselves.
• The call disconnects after job acceptance. Continue tracking the durable job and expose its status after authentication. Assert that disconnection neither marks it resolved nor silently cancels it.

Frequently asked questions

When should the avatar offer a human?

Honor an explicit human request. Also offer escalation for policy exceptions, repeated lookup failures, or unresolved ambiguity. Encode these triggers in application policy rather than asking the model to invent a confidence threshold. The trigger starts an offer; consent, ticket creation, queue admission, and human connection remain distinct steps.

Can a model approve or execute a refund?

The model cannot authorize a refund. It can recognize the request and invoke a permitted application operation. Your backend must check identity, order ownership, permissions, and refund policy before any financial action. Report success only from the payment system’s confirmed result, never from a conversation or ticket label.

Does an always-available avatar mean 24/7 human support?

No. Automated availability, staffed hours, and current queue availability are separate facts. Publish the hours you actually operate and offer asynchronous intake outside them. Neither this architecture nor the presence of an avatar establishes faster resolution or higher containment; those require evidence from your own support operation.

Should the handoff include the entire transcript?

Prefer a consented, minimized summary with facts separated from customer claims. RTA’s optional signed transcript webhook arrives after the call, so it is not the source for a live handoff. Disabling it does not establish zero retention: operational logs can contain conversation text. Review the privacy policy and define access, deletion, and retention for your own support records.

• Realtime Avatar privacy policy and retention boundaries
• Connect the conversation layer with the quickstart
• Review current Realtime Avatar plans