Build a Personal AI Assistant for Daily Planning with Voice and an Avatar

To build a personal AI assistant for daily planning, let your application assemble the user's schedule, constraints and unfinished work, then use a voice-and-avatar conversation to help them choose a workable plan. Realtime Avatar provides the managed spoken interaction and character video; your backend still owns calendar access, persistent memory and any action that changes the user's day. Start with a read-only planning conversation before giving the assistant permission to edit events.

Give the user one useful reason to open the call

A generic assistant can promise to help with anything and leave the user wondering what to ask. A daily planner has a clearer job: explain today's constraints, identify a conflict, and help the person choose what to do next. Imagine a user who has a project deadline, two meetings and a school pickup. The conversation should reveal that the afternoon is already full, not produce an impressive but impossible list of tasks.

Keep a visible plan beside the avatar. Show the date, time zone, fixed commitments and proposed changes. Someone should be able to inspect a time or correct a mistaken assumption without waiting for the character to finish a monologue. Voice is useful for explaining why an apparently free hour is not actually available. The screen is useful for seeing exactly which hour the assistant means. Neither needs to replace the other.

• The product idea: give your personal agent a face

Build a small planning snapshot, not a copy of the user's life

Resolve the signed-in account on your backend and request only the provider access the current feature needs. For a first release, that may be calendar availability plus the tasks the user explicitly selected. Do not forward email credentials, refresh tokens, full inboxes or private meeting attachments into the conversation context. A tool argument naming an account must never override the identity established by the user's authenticated session.

• Fixed commitments: event identifiers, start and end times, the relevant time zone, and whether an event can be moved.
• Chosen priorities: the tasks the user wants to discuss, with deadlines and effort estimates labelled as estimates.
• Human constraints: travel buffers, working hours and unavailable periods the user has actually supplied, not personal facts inferred from a voice or appearance.
• Freshness: when each source was read and whether a provider refused access or returned incomplete information.
• Open decisions: the few conflicts or tradeoffs this call should resolve, rather than an unbounded conversation history.

An unavailable calendar is not an empty calendar. If authorization has expired or a provider request fails, show that the schedule could not be checked and avoid recommending the unverified time as free. Keep raw source data separate from the assistant's interpretation so the UI can show the actual event even if the spoken summary is wrong. A snapshot becomes stale; reread the relevant event before committing a change.

Resolve tomorrow and next Friday before changing a calendar

A spoken instruction such as 'move it to tomorrow morning' is not yet an executable calendar update. Confirm the event, the user's intended date, the local time and the time zone. Travel and daylight-saving changes make a remembered offset an unreliable substitute for a named zone and the provider's actual temporal rules. Recurring events also need an explicit choice between one occurrence and the series.

The iCalendar standard illustrates why these distinctions matter. RFC 5545 distinguishes UTC, floating local time and time tied to a zone definition. Its Time Zone Identifier section warns that failing to include and follow the applicable VTIMEZONE definitions can produce inconsistent interpretations of local time. Follow the calendar provider's own contract; do not assume every API represents events as iCalendar or that converting every value to a timestamp preserves recurring wall-clock intent.

• RFC 5545, including Time Zone Identifier and date-time semantics

Connect the planning snapshot to the live conversation

Create or select an original character, then follow the SDK quickstart for the server connect endpoint and client call. Keep the platform API key on the server. Server-authored instructions should describe the planning role, the need to distinguish facts from suggestions, and the rule that changes require your application's confirmation flow. Supply the focused snapshot through the documented context interface instead of inventing extra fields in the strict session request.

RTA tools execute in your application, not in a hosted RTA calendar connector. A read-only availability tool can call your authenticated backend and return a small result. If an operation may exceed the documented 2.5-second tool-result deadline, have your backend record the job and return an acknowledgement promptly; show the real result when it arrives. That acknowledgement must say the work is pending, not that an event has already moved. A timeout or stopped call does not establish that an external action was cancelled.

Choose the conversation architecture before integrating providers. RTA's managed session includes speech recognition, turn taking, language-model response generation, speech synthesis and avatar video. Your backend can supply tool results and steer text responses. Bringing an external language-model endpoint requires separate workspace approval and a host allowlist. That is not the same as streaming your own speech provider's live audio into RTA; arbitrary live audio input is not a public SDK interface.

• Build the first call with the SDK quickstart
• Server-owned session instructions and context
• Application tools, deadlines and server steering

Remember decisions, not every sentence

A useful next-day check-in starts with what the person chose to keep: a deferred task, a corrected working-hours preference or a question that remains open. Your application decides what is stored, how long it remains, and how users can inspect, correct or delete it. Do not turn a tentative statement such as 'maybe I could start at seven' into a permanent preference without confirmation. The next RTA session does not automatically inherit a persistent personal memory.

Separate these product memories from operational logs. Optional signed transcript delivery is one data flow; private session diagnostics can contain conversation text and have a 30-day storage lifecycle. Turning off the transcript webhook is not a zero-retention mode. Review the actual privacy and provider policies before including sensitive calendar content, and avoid using diagnostic logs as a convenient substitute for a user-controlled memory store.

Test whether the conversation improves the plan

Compare text, voice and avatar versions using the same schedule and task data. Look for a correct next step, detected conflicts and the effort needed to correct a mistake. Session length and an appealing character are not evidence that the plan is better. Include the cost of connected time, external tools and failed attempts when deciding whether the visual format earns its place.

• Expired calendar access: the interface reports unavailable data instead of free time.
• A changed event during the call: the server detects the stale proposal before writing.
• A repeated confirmation: the same operation does not create duplicate calendar changes.
• A time-zone change or recurring event: the user sees the exact affected occurrence and local time.
• An interrupted or disconnected call: the plan survives, and the backend reports the actual operation status rather than assuming rollback.
• Microphone access denied: the person can still inspect and edit the plan through the application's non-voice interface.

Developer FAQ

Does Realtime Avatar include a calendar or email assistant?

No. It supplies the live character conversation. Your application integrates calendar and email providers, handles account permissions and executes actions. Keep the first planning release read-only if changing events is not necessary to prove its value.

Can an existing personal agent keep running after the call ends?

Yes, if that work is implemented and persisted in your own backend. The avatar call is a temporary interface, not the lifetime of the task. Store a job identifier and show its actual status when the user returns; do not imply that RTA hosts the background agent.

Will the assistant remember the user tomorrow?

Only through the context your application chooses to retain and supply. Give users a way to correct saved preferences, and rebuild the next planning snapshot from current data rather than replaying a stale plan as fact.

Should every personal assistant have a video avatar?

Not necessarily. A quick calendar lookup may be better as text. Start with a moment that benefits from discussion, such as resolving competing priorities, and compare it with simpler interfaces. RTA gives you an interaction option, not proof that video improves productivity or trust.

• Review current plans and session costs
• Read the privacy and retention policy