Designing a Personal Agent Users Can Understand, Correct and Approve

A personal agent approval and correction interface should make three things unmistakable: what the agent proposes, what the user authorizes, and what actually happened. Build a conversational interface for informed delegation: let people question the plan, change its scope, approve a specific action and inspect the result. A face can host that conversation; it cannot prove trustworthiness.

The patterns below are recommendations for your application, not built-in Realtime Avatar approval features. Our earlier personal-agent article explores why someone might want to call an agent. Here, the question is narrower: what should happen between “could you move my appointment?” and “it has been moved”?

• The broader idea: Your personal agent deserves a face

Make the proposed change inspectable

Imagine a user asking to move a bicycle repair appointment to Friday afternoon. Before requesting approval, your app should show the current booking and the exact replacement: date, time, time zone, repair shop and any fee. State whether the operation updates one booking or requires cancelling and creating another; those have different failure modes.

The avatar might explain, “Friday at 3 p.m. is available, with the same shop and no change fee.” Keep those details visible while the user asks questions. If they correct it to 4 p.m., replace the proposal rather than treating the correction as permission to proceed. Recheck availability before presenting the revised choice.

• Proposed means nothing has been submitted. Offer Edit, Approve this change and Keep current booking.
• Approved means the user authorized that proposal. Execution may still be pending or rejected.
• Verified means the booking provider confirmed the specific change. Show its evidence, not just the agent’s statement.

Bind approval to a version, not a conversation

We recommend storing an app-owned proposal version with the booking identity, exact change, relevant provider version and approval expiry. Bind the authenticated user’s approval to that version. Immediately before execution, your backend should check ownership, current permission, expiry and whether the approved facts still hold. A prompt saying “ask first” is not an authorization check.

If Friday’s slot disappears, the fee changes or someone edits the booking elsewhere, invalidate the approval and explain what changed. Do not silently choose the nearest alternative. Use conditional provider updates where available; a read followed by an unconditional write still leaves a race. If the provider cannot enforce the approved conditions, do not promise that it can.

Keep the execution scope narrow: this booking, this change, this attempt. Reading availability is not permission to cancel. Approval for rescheduling is not permission to send an additional message or modify unrelated bookings.

Keep speech interruption separate from action cancellation

Realtime Avatar supports conversational interruption, but stopping speech does not cancel an external booking request. If a user says “wait” after submission, your app should distinguish “I stopped speaking” from “the change was cancelled.” Request cancellation only where the provider supports it, and report whether it actually succeeded.

Client tools have a 2.5-second deadline. Their abort signal is cooperative: a handler that ignores it can still commit a side effect after the platform discards its result. Our recommendation is to return a truthful acknowledgement for accepted background work, track it in your backend and deliver its eventual status on screen. An acknowledgement is not a completion receipt.

• Tool calling: execution ownership, deadlines and cooperative aborts

Give every action an honest receipt

We recommend an app-owned receipt containing the approved proposal version, submission time, provider reference, verified booking details and remaining work. Preserve it after the call ends. Keep exact values in the interface; a generated spoken summary should not be the authoritative record.

• Confirmed: the provider reports the booking at the approved time. Link to the updated booking.
• Failed: the provider explicitly refused the change. Explain the refusal and offer a fresh proposal.
• Unknown: the request timed out without a conclusive result. Say that the booking may have changed, then reconcile before retrying.
• Partial: the booking changed but a separate notification failed. Report each outcome rather than calling the whole task successful.

Use provider-supported idempotency and durable attempt records to prevent duplicate execution. Reconcile unknown results against authoritative provider state. RTA’s optional post-call tool history is not a complete execution ledger: the documentation notes that interrupted tool rounds can be absent even though they ran.

Use conversation to support judgment, not obtain agreement

In a 2021 experiment with 199 participants, Buçinca, Malaya and Gajos found that cognitive forcing interventions reduced overreliance compared with simple explainable-AI approaches. The strongest interventions also received the least favorable subjective ratings. This was an AI-assisted decision-making study, not an avatar experiment or evidence that a face increases trust.

• Buçinca et al.: To Trust or to Think (2021)

Our design recommendation is selective friction: emphasize changed facts and consequential choices instead of asking for approval constantly. Disclose that the agent is AI. Make refusal and correction as accessible as approval. Avoid countdowns, guilt, flattering appeals or an avatar that celebrates before execution. A calm explanation should help someone disagree, not pressure them to consent.

Where Realtime Avatar fits

RTA manages speech recognition, language-model responses, speech synthesis and video. Your server supplies instructions and relevant context; your application owns persistent memory, permissions, approval interfaces and action execution. Tools run in your page or backend, not a hosted RTA executor. Put consequential writes behind backend authorization.

An external language-model endpoint requires workspace approval and a host allowlist. The public SDK does not accept arbitrary live audio from your own speech service. Start with the documented call integration, then build and test the appointment approval flow around your existing booking service.

• Calls: server-owned instructions and context
• Build your first avatar call

Frequently asked questions

Can spoken approval authorize an action?

Your app can implement it, but do not treat any “yes” as approval. Associate the response with one current proposal, resolve ambiguous speech and authenticate the user separately. For consequential changes, we recommend an explicit, accessible confirmation control. A familiar voice or face is not evidence of identity.

What should a correction change in persistent memory?

We recommend separating “use 4 p.m. this time” from “remember that I prefer late appointments.” Your application should let users inspect and remove saved preferences and decide which context enters future calls. RTA does not automatically give a new session persistent personal memory.

Does disabling transcript delivery mean no conversation text is retained?

No. Optional transcript delivery is separate from operational diagnostics. Private session logs can contain conversation text and have a 30-day storage lifecycle; other records and provider processing follow separate rules. Review the privacy policy and minimize sensitive booking details supplied to the call.

• Realtime Avatar privacy and retention policy

How should developers test this interface?

We recommend testing a changed fee after approval, an expired proposal, revoked permission, duplicate submission, interruption after dispatch and a timeout after provider success. Ask users to identify what is proposed, authorized and confirmed. Judge comprehension and execution correctness separately from how likable they find the avatar.