Add an Android voice companion for screen-off Codex conversations

[comfuture]

Summary

Add an Android-first, voice-only Codori companion that can keep a thread-scoped Codex realtime conversation active while the phone is locked or the companion is backgrounded.

The companion should not wrap or replace the existing Nuxt application. It should move only the browser-owned realtime media boundary into a native process: native microphone capture, RTCPeerConnection, remote audio playback, the oai-events data channel, and lifecycle-safe reconnect/teardown. Codori server and Codex app-server must remain responsible for authentication, thread context, realtime call creation, Codex handoff, tools, sandboxing, and approval policy.

Priority: P2 — native/background voice follow-up

Depends on: #85 landing the browser WebRTC/realtime foundation. Authenticated use outside a trusted local/private network also depends on #77.

Related: #81 and #82 defined the browser transport and push-to-talk UX. This issue intentionally takes native/background listening out of their non-goals without reopening or replacing the web implementation.

Background

PR #85 implements an experimental browser-owned WebRTC path for Codex realtime voice. That path is suitable while the page is foregrounded, but mobile browsers may suspend microphone capture, JavaScript, audio playback, or network activity when the screen locks. Installing the site as a PWA or placing it in a plain WebView does not establish a reliable background-microphone contract.

The desired follow-up is a small companion application for hands-free or "pocket" use:

• the existing Codori web app remains the full project/thread UI
• a native app owns only the live audio and WebRTC lifecycle
• SDP signaling still passes through Codori to Codex app-server
• spoken work still enters the selected Codex thread and follows the existing tool, sandbox, approval, and persistence behavior
• locking the phone does not silently move the realtime connection back into a suspended WebView

Android explicitly supports continued microphone capture from a foreground service for recorder or communication-style use cases, but it requires a declared microphone foreground-service type, the appropriate permissions, and a user-visible foreground-service notification. The microphone service must be started while the app is eligible to use the while-in-use microphone permission; it cannot be created opportunistically after the app is already backgrounded.

Primary platform references:

• Android foreground-service type microphone: https://developer.android.com/develop/background-work/services/fgs/service-types#microphone
• Android foreground-service launch requirements and background-start restrictions: https://developer.android.com/develop/background-work/services/fgs/launch
• Android foreground-service behavior and required user-visible notification: https://developer.android.com/develop/background-work/services/fgs
• iOS AVAudioSession background recording is a possible later platform path, but it is not part of the Android-first implementation: https://developer.apple.com/documentation/avfaudio/avaudiosession/category-swift.struct/record

Current implementation facts

Observed at PR #85 commit a2cdb49fa96477c2b88f9ccc5e8e93377f258dee:

• useRealtimeConversation.ts is explicitly browser-owned. Its environment uses navigator.mediaDevices.getUserMedia(), RTCPeerConnection, and an HTMLAudioElement.

• The controller adds the microphone audio track, creates an oai-events data channel, creates and sets an SDP offer, then sends:

  {
    "threadId": "<thread-id>",
    "outputModality": "audio",
    "version": "v3",
    "transport": {
      "type": "webrtc",
      "sdp": "<native-or-browser-offer>"
    }
  }

  through thread/realtime/start.

• The controller applies thread/realtime/sdp as the remote SDP answer and consumes thread/realtime/started, transcript delta/done, error, and closed notifications over the existing Codori RPC connection.

• ThreadRealtimeStartTransport.ts requires only a WebRTC SDP offer. The generated contract does not require that the offer originate in a browser.

• codex-rpc.ts already centralizes JSON-RPC requests, notifications, connection state, and safe default responses for approval/input requests.

• http-server.ts exposes project and projectless-chat app-server WebSocket proxies at /api/projects/:projectId/rpc and /api/chats/:chatId/rpc.

• The managed server uses @openai/codex ^0.145.0, and realtime voice remains default-off behind Codori's realtimeVoice.enabled / --experimental-realtime-voice gate.

• Add push-to-talk voice coding controls to the chat composer #82 deliberately excluded background listening and native companion applications. Its browser push-to-talk state machine remains the behavior and safety reference for thread/session ownership, transcript reconciliation, and cleanup.

• PR feat: add realtime voice conversations #85 is still open. The implementation must re-check the merged code, installed @openai/codex version, generated realtime contract, and active transport behavior before starting native work.

Problem statement

Today the browser owns every resource that must remain active for a voice conversation: microphone permission and capture, RTCPeerConnection, remote playback, the app-server control WebSocket, and the local session generation. When the browser is suspended, Codex app-server may remain running and the thread may remain persisted, but the live audio/control path is not reliable.

A PWA or WebView-only wrapper would retain the same browser ownership and suspension risks. Conversely, a standalone native app that calls OpenAI directly would bypass the existing ChatGPT-authenticated Codex app-server path, duplicate authentication and thread behavior, and create a new API-key/security surface.

The missing capability is therefore a narrow native media client that speaks to the existing Codori/app-server boundary without becoming a second full Codori client.

Goal

Provide an Android voice companion that a paired/trusted user can hand an existing Codori thread to, explicitly start, mute, resume, and stop, and continue using with the screen locked.

The companion must own microphone capture and WebRTC natively, while Codori and Codex app-server continue to own the realtime call, selected thread, Codex handoff, sandbox/tools, and persisted work. Existing browser voice and text workflows must continue to work when the companion is not active.

Scope

This issue should include:

• An Android-first native companion package/application in the Codori repository, with the framework choice documented before implementation.
• A minimal voice-focused UI: configured Codori server, connection/thread identity, session state, microphone mute/resume, output mute, explicit stop, transcript/status feedback, and actionable errors.
• A deliberate thread handoff from the Codori web UI, such as an authenticated deep link or QR-assisted flow. Handoff data may include a server profile identifier and project/chat/thread identifiers, but must not embed reusable secrets.
• A native WebRTC adapter that owns the microphone track, remote audio track, RTCPeerConnection, oai-events data channel, local SDP offer, remote SDP answer, and ICE/connection state.
• Reuse of the existing Codori project/chat RPC boundary, or a narrowly scoped Codori voice gateway if direct companion use of the general RPC proxy cannot meet lifecycle, authentication, or ownership requirements.
• The same V3 thread/realtime/start semantics used by PR feat: add realtime voice conversations #85: outputModality: "audio", the selected threadId, WebRTC transport, app-server startup context, and automatic Codex handoffs. Do not submit final speech as a duplicate turn/start.
• Subscription before start to thread/realtime/started, thread/realtime/sdp, transcript delta/done, error, closed, and relevant Codex activity notifications.
• Android microphone permission plus a microphone foreground service started from an eligible visible state. The ongoing notification must make capture visible and provide at least mute and stop actions.
• Screen-lock/background lifecycle handling that keeps the native audio/WebRTC/control connection alive while the foreground service is active.
• Deterministic teardown on explicit stop, permission loss, service stop, app force-stop, unrecoverable RPC/WebRTC failure, thread ownership loss, and partial startup failure.
• Bounded reconnection across temporary Codori RPC disconnects and ordinary Wi-Fi/mobile-network changes. Capture must fail safe: no stale client or replaced session may continue transmitting.
• Audio focus, phone-call interruption, route-change, wired/Bluetooth device, and output-resume handling appropriate for a conversational audio session.
• One authoritative realtime owner per thread. Starting the companion must either replace or reject an existing browser voice session deterministically, and both clients must surface ownership loss without stale audio or transcript leakage.
• A safe policy for app-server approval, permission, elicitation, and user-input requests received while the voice-only companion owns the originating RPC connection. The first version must never convert spoken assent into an automatic sensitive approval. Unsupported prompts should fail closed and direct the user to the full Codori UI.
• Secure persistence for Codori server profiles and paired-device material using Android platform storage. No OpenAI API key or ChatGPT bearer may be requested, stored, logged, or placed in handoff URLs.
• Unit, integration, and real-device instrumentation/smoke coverage for native session state, SDP signaling, background audio, notification actions, ownership, reconnection, and teardown.

Non-goals

This issue should not include:

• Replacing, removing, or packaging the complete Nuxt Codori application.
• A PWA-only or WebView-only solution where browser JavaScript still owns microphone capture and RTCPeerConnection.
• Direct OpenAI Realtime API connectivity, OpenAI API-key entry/storage, or reading Codex credentials from the host.
• Proxying raw microphone audio through the Codori HTTP/WebSocket server. Raw media should remain WebRTC media between the native peer and the realtime service.
• Reimplementing Codex tools, shell/file operations, sandbox policy, turn execution, or transcript persistence in the native app.
• Automatically approving command, file-change, permission, MCP elicitation, or user-input requests from voice alone.
• A complete mobile project browser, chat transcript, plan/tool renderer, diff viewer, terminal, file explorer, or settings application.
• Wake-word activation, indefinite ambient monitoring, stealth capture, OS-global hotkeys, or starting microphone capture without an explicit user action and visible system indicator.
• Storing or uploading audio recordings after the live WebRTC session ends.
• iOS implementation, App Store/Play Store release automation, push notifications, or production mobile distribution. The Android design should avoid needless platform lock-in, but iOS is a separate follow-up.
• Solving general multi-user/RBAC behavior. Remote production use should build on Add device pairing and authenticated browser sessions for remote access #77's trusted-device boundary.

Suggested direction

1. Freeze a narrow companion contract

Before choosing UI framework or WebRTC library, record the exact merged PR #85 payloads and notifications as versioned fixtures. Keep the native adapter limited to:

• Codori server/session authentication
• resource identity (projectId or chatId) and threadId
• initialize/capability/start/stop requests
• SDP and realtime lifecycle/transcript notifications
• required server-request fail-closed responses

Do not manually copy the entire generated app-server protocol into the mobile application. Isolate the experimental V3 adapter so a future @openai/codex update can be refreshed and tested in one place.

Start with the existing project/chat RPC proxy. If maintaining a general app-server socket in a foreground service exposes too broad a surface or makes cross-client ownership/request routing ambiguous, add a small authenticated Codori-owned voice gateway rather than making the companion a full app-server client.

2. Implement native WebRTC ownership

Use a native Android WebRTC implementation, not a WebView RTCPeerConnection.

1. Acquire RECORD_AUDIO from a visible activity after explicit user intent.
2. Start the microphone foreground service while the app is eligible to use the permission.
3. Create the native peer connection, configure remote playback, add one microphone audio track, and create the oai-events data channel.
4. Create/set the local offer and send the resulting SDP through Codori to thread/realtime/start.
5. Apply thread/realtime/sdp as the answer and require both the expected V3 start notification and a usable peer state before reporting the session ready.
6. Route remote audio through a conversational AudioManager configuration and surface route/interruption state.
7. Keep one monotonically increasing local session generation and ignore all stale callbacks from replaced attempts.

The browser controller and tests in PR #85 are the behavioral oracle for SDP ordering, connection timeout, transcript reconciliation, stale-generation rejection, and idempotent cleanup.

3. Make background capture explicit and observable

Use a foreground service with the Android microphone service type and the required current-platform permissions. The service notification should identify Codori voice as active and expose mute and stop actions.

Do not start or resume capture silently from a background receiver. After process death, force-stop, revoked permission, or an expired session, require a new foreground user action. A short network interruption may reconnect only while the same explicitly active foreground session and notification remain valid.

Treat an incoming call, audio-focus loss, output route removal, or unrecoverable peer failure as a visible paused/error state. Never leave the system microphone indicator active after the Codori session is terminal.

4. Hand off threads without handing off secrets

Add an "Open in Voice Companion" action to the web voice surface after the thread has a stable ID. Prefer an app link/deep link that references a previously paired server profile and non-secret project/chat/thread IDs.

For development before #77 lands, permit an explicitly documented trusted local/private-network profile. Do not normalize unauthenticated remote access as the production design. Once #77 is available, extend its pairing/session contract for a native device credential stored in Android secure storage; do not scrape or reuse a browser HttpOnly cookie through a WebView.

5. Define cross-client and approval behavior

Exercise browser-to-native, native-to-browser, duplicate-native, and reconnect races against one thread. Prefer app-server's upstream replacement/closed behavior if it is deterministic; otherwise add the smallest server-side lease/generation coordinator necessary.

The voice-only companion should show Codex progress and transcript status, but it should not become a hidden approval client. If a sensitive approval or unsupported input request occurs, stop or mute transmission as appropriate, return the established safe response, and tell the user to continue in the full Codori UI. A future issue may add an authenticated approval handoff after its request-lifetime and ownership semantics are designed.

Open questions

• Which native stack gives the smallest maintainable Android artifact while providing a real native WebRTC peer and foreground service: Kotlin/Compose, Flutter with native audio/WebRTC ownership, or another thin shell?
• Can the existing general app-server RPC proxy remain connected reliably from the Android foreground service, or should the server expose a smaller Codori-owned voice WebSocket?
• Does the merged upstream/app-server behavior provide deterministic cross-client realtime replacement notifications, or is a Codori per-thread lease required?
• Which app-server server requests can occur during V3 automatic handoff, and which safe response allows the user to continue later in the web UI without leaving a hung turn?
• What minimum thread handoff UX is sufficient before Add device pairing and authenticated browser sessions for remote access #77 provides a production native pairing credential?

Resolve these with a short implementation spike and record the result in the PR; do not silently choose a WebView because it is faster to scaffold.

Acceptance criteria

• The existing Codori web app remains independently usable and is not embedded as the companion's primary runtime.
• On a supported Android device, an explicitly started voice session uses native microphone capture and a native RTCPeerConnection; WebView/browser media APIs are not the owner.
• The native offer is sent through Codori/app-server using the selected thread, outputModality: "audio", version: "v3", and transport: { type: "webrtc", sdp }.
• The companion creates the oai-events data channel, applies the app-server SDP answer, receives remote audio, and reconciles user/assistant transcript events without duplicates or cross-thread leakage.
• Spoken work is delegated into the same Codex thread through app-server automatic handoff. The companion never sends the final speech transcript as a duplicate turn/start.
• Locking the screen for at least ten minutes during an active foreground session preserves two-way audio, transcript delivery, and Codex handoff on the documented reference device/API levels.
• The foreground-service notification remains visible for the entire background microphone session and its mute/stop actions take effect promptly.
• Unlocking/reopening the app shows the same active session and correct microphone/output/connection/thread state.
• Temporary Wi-Fi/mobile-network changes either reconnect within a documented bound or stop capture and report a terminal error; they never leave an unowned microphone track transmitting.
• Explicit stop, notification stop, ownership replacement, RPC failure, peer failure, permission revocation, service destruction, app force-stop, and partial startup failure release microphone/audio/WebRTC/RPC resources exactly once.
• A browser session and native companion cannot both believe they own the same thread's active realtime session. Replacement or rejection is visible and stale events cannot mutate the new owner.
• App-server approval, permission, elicitation, and user-input requests are never auto-approved from speech. Unsupported companion flows fail closed and direct the user to the full UI.
• Handoff URLs/QR payloads and logs contain no OpenAI API key, ChatGPT bearer, reusable pairing secret, microphone bytes, or raw persisted audio.
• Existing browser realtime voice, ordinary text chat, Codex turn/tool/approval behavior, and server management APIs remain functional when the companion is disconnected.
• Native unit/integration tests, Android lint/build checks, focused Codori server/client tests, and repository pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass.

Suggested validation

Automated coverage should include:

• exact V3 start payload and notification decoding from versioned fixtures
• offer/answer ordering, data-channel creation, remote-track attachment, and timeout
• session generation, duplicate start, cross-client replacement, and stale-event rejection
• transcript delta/done reconciliation for both roles
• foreground-service start prerequisites, notification mute/stop actions, and permission denial/revocation
• audio focus/interruption and wired/Bluetooth route changes where testable
• RPC disconnect, bounded reconnect, network switch, peer failure, and partial-start cleanup
• safe handling of every app-server server-request kind observed during a realtime Codex handoff
• deep-link parsing and rejection of unknown servers, malformed IDs, embedded secrets, and unauthenticated remote origins
• regression coverage for the existing browser controller and Codori RPC proxies

Real-device smoke test:

1. Start Codori with experimental realtime voice enabled and Codex logged in through ChatGPT; do not provide an OpenAI API key to the companion.
2. Open a stable thread in the Codori web UI and hand it to a paired/trusted Android device.
3. Start the native voice session while the app is visible, confirm the foreground-service notification, speak a repository question, and verify the same thread receives the Codex handoff.
4. Lock the device, put it in a pocket for at least ten minutes, speak follow-ups, and confirm two-way audio and thread activity continue.
5. Use notification mute/resume and stop while locked; confirm the microphone indicator and remote audio match the requested state.
6. Repeat across a Wi-Fi-to-mobile transition, temporary server disconnect, wired headset, Bluetooth headset, incoming-call interruption, and screen unlock.
7. Start browser voice for the same thread during the native session and verify deterministic ownership replacement/rejection with no stale capture.
8. Trigger a Codex operation that requires approval and confirm the companion does not auto-approve it or hang indefinitely.
9. Force-stop the app and revoke microphone permission; confirm every resource is released and reconnect requires a new foreground user action.

Record the Android versions, device models, network path, Codex version, and observed lifecycle/connection states in the implementation PR.

Documentation follow-up

Document:

• the voice-only companion architecture and why it is not a PWA/WebView wrapper
• Android permissions, foreground-service notification, background behavior, and battery/network limitations
• how to pair/configure a Codori server and hand a thread to the companion
• how mute, stop, ownership replacement, reconnect, and approval-required states behave
• the ChatGPT-authenticated, API-key-free app-server boundary
• supported Android/API levels and real-device validation matrix
• the explicit non-goals for wake words, stealth/ambient recording, iOS, and full mobile Codori UI