diff --git a/packages/webxr/README.md b/packages/webxr/README.md index 373965a..035c5cf 100644 --- a/packages/webxr/README.md +++ b/packages/webxr/README.md @@ -35,8 +35,12 @@ framebuffer scaling or fixed foveation. The kitchen-sink fixture also opens the lime status window with primary and secondary quota windows, context state, and the initial action registry. Append `&blend=alpha-blend` or `&blend=additive` to preview the two -development-only agent contrast treatments. These are visual fixtures, not -claims that a camera or optical display is active. +development-only agent contrast treatments. The fixture also shows both +application-rendered hand outlines. For alpha blend, add +`&background=bright`, `&background=dark`, or `&background=textured` to compare +the transparent composition against deterministic surrogate environments. +These are visual fixtures, not claims that a camera or optical display is +active. Vite proxies `/api` HTTP and WebSocket traffic to `http://127.0.0.1:4310` by default. Point it at another running Codori server @@ -64,13 +68,14 @@ Users sensitive to motion or flicker should enable reduced effects before enteri Status window: - The verified `menu` component on the left `htc-vive-focus` input profile toggles the window. Codori does not guess extra gamepad indices on other profiles, and WebXR-reserved app/system buttons may be absent. -- With a tracked left hand and no active left controller, hold the raised back-of-hand pose for `450 ms`. Lowering it for `180 ms` dismisses the window; short pose/tracking gaps use hysteresis and a `300 ms` tracking-loss grace. +- With a tracked left hand and no active left controller, look toward the raised back-of-hand pose and hold it for `700 ms`. The gaze gate uses the headset's forward direction because standard WebXR does not expose portable eye-gaze tracking. An open hand window dismisses after either a continuous `18 cm` natural lowering trajectory held past the lower endpoint for `320 ms`, or a clear palm-facing turn held for `560 ms`. Edge-on rotation, implausible wrist jumps, and temporary tracking loss remain inert. As the right index approaches the action area, the last stable window pose and dismissal state are protected; direct touch begins only when the fingertip joint radius reaches the panel surface and the touched row brightens for feedback. - A controller-opened window remains open until reinvoked, an action is selected, or the left grip is held below the lowering threshold for `250 ms`. -- Placement uses the left grip or WebXR `wrist` joint plus a viewer-facing vertical offset. WebXR exposes no elbow joint, so this is a wrist/grip-and-view approximation rather than full forearm tracking. -- Right controller rays and direct controller contact can activate actions. Tracked hands require direct `index-finger-tip` contact; remote pinch/ray activation is rejected for status actions. +- Hand-opened placement uses the WebXR `wrist` world position plus an `18 cm` back-of-hand rise, so the window follows the left arm's forward/back, lateral, and vertical motion instead of being normalized to a fixed viewer distance. Controller placement retains its directional `0.4 m` reading distance. The physical surface is `0.24 m × 0.32 m`, exactly one third of the previous window and text scale. If the wrist is temporarily lost, the window stays at its last stable world pose instead of jumping away or closing. WebXR exposes no elbow joint, so this remains a wrist approximation rather than full forearm tracking. +- Presentation and activation use separate states. Actions remain inert while the window emerges and for an additional `180 ms` arming grace. The gate snapshots held presses and fingertip overlaps at the armed boundary; each input channel must cross neutral/release before a new transition can activate anything. +- Only a right controller ray/direct-controller contact click or fresh right `index-finger-tip` contact can activate an action. Hand contact uses an oriented-box-versus-fingertip-sphere intersection against the real panel surface with no forward tolerance; the hand outline's outer fingertip radius matches the same WebXR collision radius. Left-hand input, screen/gaze action input, native or synthesized hand pinch, held select, and stale contact are rejected. One press/contact transition can activate at most once. - When no exposed mapped menu component or gesture-eligible tracked left hand can invoke status, an in-canvas bottom-right `Menu` is shown. This includes right-hand-only tracking and controllers whose app/system button is reserved or unmapped; their target ray can select the fallback. A left controller takes precedence over the left-hand gesture, so an unmapped left controller keeps the fallback available for ray selection. A DOM-overlay button mirrors it when the optional DOM Overlays feature is granted. Both disappear when a mapped menu control or eligible left-hand gesture becomes usable; screen/gaze can open the menu but cannot bypass `controller-or-touch` action policy. -The status view uses a translucent pale lime treatment distinct from cyan panes. It shows the authoritative primary and secondary Codex quota windows, localized reset times, current-thread context remaining only when known, connection/voice state, pane count, and thread/workspace identity. Sparse `account/rateLimits/updated` buckets merge by `limitId` into the last `account/rateLimits/read` snapshot; unknown quota or context is labeled unavailable rather than rendered as zero/full. +The status view uses a translucent, saturated lime treatment distinct from cyan panes, with rounded outer corners and the same soft canvas-border glow language as panes and transcript bubbles. It shows the authoritative primary and secondary Codex quota windows, localized reset times, current-thread context remaining only when known, connection/voice state, pane count, and thread/workspace identity. Sparse `account/rateLimits/updated` buckets merge by `limitId` into the last `account/rateLimits/read` snapshot; unknown quota or context is labeled unavailable rather than rendered as zero/full. Initial actions are `Passthrough`, `Recenter workspace`, the live voice/resume-audio action, `Reduced effects`, and `Exit immersive`. Each registry item carries a stable id, state, availability/disabled reason, callback, and input policy so additional actions can be added without changing the status surface architecture. @@ -86,16 +91,37 @@ Controller: Tracked hand: +- WebXR exposes articulated joint poses and radii; it does not guarantee a compositor-rendered hand. Codori therefore draws a lightweight connected outline for each tracked hand in opaque VR, hides it in passthrough or when tracking is lost or a same-handed controller is active, and excludes every outline primitive from raycasts and collisions. Passthrough hides only the synthetic outline; joint tracking and hand input remain active. - direct `index-finger-tip` contact moves a nearby pane from any non-actionable point -- index/thumb pinch remotely grabs a distant pane; opening the pinch releases it -- visible top/bottom triangles are direct fingertip scroll controls. Scrolling starts slowly, accelerates smoothly to a cap while contact remains, and stops immediately on leave or tracking/source loss +- index/thumb pinch remotely grabs a distant pane from the physical fingertip midpoint. Pulling the pinched hand toward the viewer accelerates only the depth component by `3.2×` within the `0.65–4.5 m` clamps. Left/right and up/down displacement continuously scales by the current pane-distance-to-hand-distance ratio so angular movement remains consistent as depth changes; a `0.25 m` minimum hand distance bounds tracking spikes without introducing a far-pane cap. Opening the pinch releases it +- visible top/bottom triangles sit inside the content area below the title bar and use expanded invisible fingertip targets. Direct contact starts slowly, accelerates smoothly to a cap while contact remains, and stops immediately on leave or tracking/source loss; a remote controller or hand-pinch cursor selection moves one line per press - remote pinch never activates status-window actions, which retain the direct-touch-only hand policy Native and synthesized primary actions are de-duplicated. Preferred hints follow the most recently used connected source independently per hand, so a valid hand can replace an unused/disconnected controller without a session restart. Competing grabs have one deterministic owner, and input-source or required-joint loss releases hover, grab, and held-scroll state. +The central agent-light orb starts voice only while voice is inactive. Its hit target is removed throughout permission, offer creation, startup, connected, and stopping states, preventing an accidental touch from ending an active session; intentional stopping remains available from the status window. + +Status-window binary actions render as track-and-knob toggles, while recenter and exit remain one-shot buttons. + +Assistant transcript bubbles use a more opaque sky-blue surface and brighter text/border treatment for passthrough readability. + +The palm-up per-hand `Menu`/Meta suggestions seen on Quest are Meta Horizon OS +trusted UI, not the Codori fallback menu. Standard WebXR exposes joint poses and +platform-defined `select*` events but no API for repositioning, freezing, +suppressing, or restyling those system icons. Codori therefore does not imitate +or intercept them, and its hand outline remains thin and non-interactive so it +does not compete with the trusted suggestion. Meta's v66 documentation +described an experimental wrist Meta-button setting, but setting names and +availability vary by Horizon OS rollout; inspect the current headset under +Movement Tracking rather than assuming a portable application control. See the +[WebXR trusted-environment requirements](https://immersive-web.github.io/webxr/), +[WebXR Hand Input](https://immersive-web.github.io/webxr-hand-input/), +[Meta native system-gesture guidance](https://developers.meta.com/horizon/documentation/native/android/mobile-hand-tracking/), +and [Meta Quest v66 release notes](https://communityforums.atmeta.com/blog/AnnouncementsBlog/meta-quest-build-v66-release-notes/1209566). + `Recenter workspace` rotates and translates one shared anchor into the current horizontal gaze at clamped eye height. The agent light, transcript, status surfaces, automatic panes, and manually moved pane-local transforms move together without reallocating pane ids. A `local-floor` reference-space `reset` schedules exactly one anchor refresh; Codori does not emulate or require a reserved platform recenter button. -`Passthrough` is a real session-mode transition, not a background toggle. When `immersive-ar` is supported, Codori ends the current session and requests the other mode while retaining the same workspace and voice runtimes/subscriptions. If the browser cannot complete that transition within the action, the entry surface explains that state and offers explicit re-entry without recreating the RPC runtime. In a non-opaque AR session, transparent renderer pixels expose the environment and room geometry plus the boundary Exit door are hidden. `alpha-blend` uses a restrained dark stipple shell outside the agent light; `additive` uses a bright magenta shape outline because dark pixels cannot occlude an optical display. An AR session reporting `opaque` is never described as passthrough. The normal projection path remains correct without WebXR Layers. +`Passthrough` is a real session-mode transition, not a background toggle. When `immersive-ar` is supported, Codori ends the current session and requests the other mode while retaining the same workspace and voice runtimes/subscriptions. If the browser cannot complete that transition within the action, the entry surface explains that state and offers explicit re-entry without recreating the RPC runtime. In a non-opaque AR session, transparent renderer pixels expose the environment and room geometry plus the boundary Exit door are hidden. `alpha-blend` uses one neutral-dark, low-opacity radial feather that fades continuously to transparent outside the agent light's flare; it contains no dots, noise, checker, stipple, or hard ring. `additive` uses a bright magenta shape outline because dark pixels cannot occlude an optical display. An AR session reporting `opaque` is never described as passthrough. The normal projection path remains correct without WebXR Layers. Immersive entry starts the realtime voice session automatically after at least `500 ms` and as soon as the workspace runtime is ready. New XR sessions reuse the voice selection and browser-only voice-instruction override saved by Settings → Voice. The selected voice is sent only when the connected server advertises it; prompt precedence remains browser override, `experimental_realtime_ws_backend_prompt` from `config.toml`, then the Codori default. Selecting the central light stops the active session and re-arms the dormant visual state; selecting it again replays the full awakening before restarting voice. A door-sized rounded `Exit` surface sits on the `10 m × 10 m` room boundary, beyond the agent light along the initial view direction. The 2D fallback remains available before immersive entry. @@ -105,7 +131,9 @@ Web Audio synthesizes a one-second agent-awakening cue whose low mechanical chor ## Panel semantics and caps -Foreground command, file-change, MCP, dynamic-tool, and web-search panels appear near the central field of view on `item/started`, update continuously from progress/output deltas, stay visible while active, dwell for one minute after terminal completion, then shrink and dispose. Four screen-space anchors each have a front/back depth pair: when a new panel reaches an occupied anchor, the existing panel eases `0.55 m` back and the new panel opens in front. Scrolling, moving, or focusing a completed panel restarts that one-minute dwell. A manually dismissed panel does not reappear when late deltas arrive. +Foreground command, file-change, MCP, dynamic-tool, and web-search panels appear near the central field of view on `item/started`, update continuously from progress/output deltas, stay visible while active, dwell for one minute after terminal completion, then shrink and dispose. Four screen-space anchors each have a front/back depth pair. New panes prefer an anchor whose projected area is clear of both automatic and manually positioned panes. If overlap is unavoidable, the new pane receives the nearer depth; when every automatic slot is full, the newest pane takes a front slot, its previous occupant moves back, and the displaced back pane remains queued as overflow rather than covering the latest information. Scrolling, moving, or focusing a completed panel restarts that one-minute dwell. A manually dismissed panel does not reappear when late deltas arrive. + +The opaque-room floor, grid, and walls render before panes without writing depth, so panels remain readable even when placed behind or below the visual room shell. Pane-to-pane depth ordering remains intact. Panel height follows the estimated wrapped output, including double-width CJK characters. Short results use a compact `0.44 m` panel and grow with their content; the previous `0.92 m` height is the maximum. @@ -122,6 +150,6 @@ Growing output follows the live tail until manual scrolling. Later deltas preser ## Current validation boundary -Automated tests cover session options/failure states, deterministic light bounds and reduced effects, transcript generation plus 30-second visibility and 250 ms scale transitions, streaming panel lifecycle plus 60-second dwell and 125 ms forced dismissal, panel retention/layout/scroll state, input ownership and source loss, shared notification adapters, `/xr/` server routing, and package builds. +Automated tests cover session options/failure states, deterministic light bounds and reduced effects, transcript generation plus 30-second visibility and 250 ms scale transitions, streaming panel lifecycle plus 60-second dwell and 125 ms forced dismissal, panel retention/layout/scroll state, input ownership and source loss, status reveal/arming/neutral transitions, right-only action activation, hand-outline geometry and lifecycle, the smooth passthrough feather contract, shared notification adapters, `/xr/` server routing, and package builds. Real headset validation is still required before making device-specific support, transition, input-component, blend-mode, or performance claims. For each controller, hand-tracking, and screen/gaze device/browser combination, record headset/device, OS, browser version, supported session modes, actual `environmentBlendMode` and `interactionMode`, optional features granted, exposed input profiles/components, app-visible buttons, target refresh rate, median frame time, sustained worst frame-time band, status anchoring/direct-touch behavior, alpha/additive contrast readability, text legibility, and a 15-minute mixed voice/tool memory observation. Browser kitchen-sink QA proves only canvas layout, colors, typography, and animation; it cannot prove headset poses, camera passthrough, additive optics, reserved buttons, or seamless session switching. diff --git a/packages/webxr/src/hand-outline-view.ts b/packages/webxr/src/hand-outline-view.ts new file mode 100644 index 0000000..303a8d3 --- /dev/null +++ b/packages/webxr/src/hand-outline-view.ts @@ -0,0 +1,315 @@ +import { + CylinderGeometry, + Group, + InstancedMesh, + Matrix4, + MeshBasicMaterial, + Quaternion, + SphereGeometry, + Vector3, + type XRHandSpace +} from 'three' + +export const HAND_JOINT_NAMES: readonly XRHandJoint[] = [ + 'wrist', + 'thumb-metacarpal', + 'thumb-phalanx-proximal', + 'thumb-phalanx-distal', + 'thumb-tip', + 'index-finger-metacarpal', + 'index-finger-phalanx-proximal', + 'index-finger-phalanx-intermediate', + 'index-finger-phalanx-distal', + 'index-finger-tip', + 'middle-finger-metacarpal', + 'middle-finger-phalanx-proximal', + 'middle-finger-phalanx-intermediate', + 'middle-finger-phalanx-distal', + 'middle-finger-tip', + 'ring-finger-metacarpal', + 'ring-finger-phalanx-proximal', + 'ring-finger-phalanx-intermediate', + 'ring-finger-phalanx-distal', + 'ring-finger-tip', + 'pinky-finger-metacarpal', + 'pinky-finger-phalanx-proximal', + 'pinky-finger-phalanx-intermediate', + 'pinky-finger-phalanx-distal', + 'pinky-finger-tip' +] as const + +export const HAND_BONE_CONNECTIONS: readonly ( + readonly [XRHandJoint, XRHandJoint] +)[] = [ + ['wrist', 'thumb-metacarpal'], + ['thumb-metacarpal', 'thumb-phalanx-proximal'], + ['thumb-phalanx-proximal', 'thumb-phalanx-distal'], + ['thumb-phalanx-distal', 'thumb-tip'], + ['wrist', 'index-finger-metacarpal'], + ['index-finger-metacarpal', 'index-finger-phalanx-proximal'], + ['index-finger-phalanx-proximal', 'index-finger-phalanx-intermediate'], + ['index-finger-phalanx-intermediate', 'index-finger-phalanx-distal'], + ['index-finger-phalanx-distal', 'index-finger-tip'], + ['wrist', 'middle-finger-metacarpal'], + ['middle-finger-metacarpal', 'middle-finger-phalanx-proximal'], + ['middle-finger-phalanx-proximal', 'middle-finger-phalanx-intermediate'], + ['middle-finger-phalanx-intermediate', 'middle-finger-phalanx-distal'], + ['middle-finger-phalanx-distal', 'middle-finger-tip'], + ['wrist', 'ring-finger-metacarpal'], + ['ring-finger-metacarpal', 'ring-finger-phalanx-proximal'], + ['ring-finger-phalanx-proximal', 'ring-finger-phalanx-intermediate'], + ['ring-finger-phalanx-intermediate', 'ring-finger-phalanx-distal'], + ['ring-finger-phalanx-distal', 'ring-finger-tip'], + ['wrist', 'pinky-finger-metacarpal'], + ['pinky-finger-metacarpal', 'pinky-finger-phalanx-proximal'], + ['pinky-finger-phalanx-proximal', 'pinky-finger-phalanx-intermediate'], + ['pinky-finger-phalanx-intermediate', 'pinky-finger-phalanx-distal'], + ['pinky-finger-phalanx-distal', 'pinky-finger-tip'], + ['thumb-metacarpal', 'index-finger-metacarpal'], + ['index-finger-metacarpal', 'middle-finger-metacarpal'], + ['middle-finger-metacarpal', 'ring-finger-metacarpal'], + ['ring-finger-metacarpal', 'pinky-finger-metacarpal'] +] as const + +export type HandOutlineJointPose = { + position: Vector3 + radius: number +} + +export type HandOutlinePose = ReadonlyMap + +const yAxis = new Vector3(0, 1, 0) +const midpoint = new Vector3() +const direction = new Vector3() +const scale = new Vector3() +const rotation = new Quaternion() +const matrix = new Matrix4() + +const handColor = (handedness: XRHandedness) => handedness === 'left' + ? '#a8efff' + : '#d8ffad' + +export const resolveHandJointRadius = (radius?: number) => + Math.max(0.001, radius ?? 0.006) + +export class HandOutlineView { + readonly group = new Group() + + readonly outerBones: InstancedMesh + + readonly innerBones: InstancedMesh + + readonly outerJoints: InstancedMesh + + readonly innerJoints: InstancedMesh + + private readonly boneGeometry = new CylinderGeometry(1, 1, 1, 8, 1, true) + + private readonly jointGeometry = new SphereGeometry(1, 10, 7) + + private readonly outerMaterial = new MeshBasicMaterial({ + color: '#061116', + transparent: true, + opacity: 0.28, + depthWrite: false, + depthTest: true, + toneMapped: false + }) + + private readonly innerMaterial = new MeshBasicMaterial({ + color: '#d8ffad', + transparent: true, + opacity: 0.52, + depthWrite: false, + depthTest: true, + toneMapped: false + }) + + private readonly trackedPose = new Map() + + private currentHandedness: XRHandedness + + constructor(handedness: XRHandedness = 'none') { + this.currentHandedness = handedness + const boneCapacity = HAND_BONE_CONNECTIONS.length + const jointCapacity = HAND_JOINT_NAMES.length + this.outerBones = new InstancedMesh( + this.boneGeometry, + this.outerMaterial, + boneCapacity + ) + this.innerBones = new InstancedMesh( + this.boneGeometry, + this.innerMaterial, + boneCapacity + ) + this.outerJoints = new InstancedMesh( + this.jointGeometry, + this.outerMaterial, + jointCapacity + ) + this.innerJoints = new InstancedMesh( + this.jointGeometry, + this.innerMaterial, + jointCapacity + ) + this.group.name = `tracked-hand-outline-${handedness}` + this.group.userData.representationalOnly = true + this.innerMaterial.color.set(handColor(handedness)) + for (const object of [ + this.outerBones, + this.innerBones, + this.outerJoints, + this.innerJoints + ]) { + object.frustumCulled = false + object.userData.representationalOnly = true + object.raycast = () => {} + } + this.outerBones.renderOrder = 20 + this.outerJoints.renderOrder = 20 + this.innerBones.renderOrder = 21 + this.innerJoints.renderOrder = 21 + this.group.add( + this.outerBones, + this.outerJoints, + this.innerBones, + this.innerJoints + ) + this.clear() + } + + get handedness() { + return this.currentHandedness + } + + setHandedness(handedness: XRHandedness) { + this.currentHandedness = handedness + this.group.name = `tracked-hand-outline-${handedness}` + this.innerMaterial.color.set(handColor(handedness)) + } + + updateFromHand(hand: XRHandSpace, allowed: boolean) { + this.trackedPose.clear() + if (allowed && hand.visible) { + for (const name of HAND_JOINT_NAMES) { + const joint = hand.joints[name] + if (!joint?.visible) { + continue + } + this.trackedPose.set(name, { + position: joint.position, + radius: resolveHandJointRadius(joint.jointRadius) + }) + } + } + this.update(this.trackedPose, allowed) + } + + update(pose: HandOutlinePose, allowed = true) { + let boneCount = 0 + for (const [fromName, toName] of HAND_BONE_CONNECTIONS) { + const from = pose.get(fromName) + const to = pose.get(toName) + if (!from || !to) { + continue + } + direction.subVectors(to.position, from.position) + const length = direction.length() + if (length < 0.0001) { + continue + } + midpoint.copy(from.position).add(to.position).multiplyScalar(0.5) + rotation.setFromUnitVectors(yAxis, direction.divideScalar(length)) + const trackedRadius = Math.min(0.007, (from.radius + to.radius) * 0.18) + scale.set(trackedRadius * 1.75, length, trackedRadius * 1.75) + matrix.compose(midpoint, rotation, scale) + this.outerBones.setMatrixAt(boneCount, matrix) + scale.x = trackedRadius + scale.z = trackedRadius + matrix.compose(midpoint, rotation, scale) + this.innerBones.setMatrixAt(boneCount, matrix) + boneCount += 1 + } + + let jointCount = 0 + for (const name of HAND_JOINT_NAMES) { + const joint = pose.get(name) + if (!joint) { + continue + } + const radius = Math.max(0.003, joint.radius) + // The outer translucent surface matches the WebXR joint radius used by + // direct-touch collision so visible fingertip contact and activation + // share one physical boundary. + scale.setScalar(radius) + matrix.compose(joint.position, rotation.identity(), scale) + this.outerJoints.setMatrixAt(jointCount, matrix) + scale.setScalar(radius * 0.58) + matrix.compose(joint.position, rotation, scale) + this.innerJoints.setMatrixAt(jointCount, matrix) + jointCount += 1 + } + + this.outerBones.count = boneCount + this.innerBones.count = boneCount + this.outerJoints.count = jointCount + this.innerJoints.count = jointCount + for (const mesh of [ + this.outerBones, + this.innerBones, + this.outerJoints, + this.innerJoints + ]) { + mesh.instanceMatrix.needsUpdate = true + } + this.group.visible = allowed && boneCount >= 4 && jointCount >= 5 + } + + clear() { + this.outerBones.count = 0 + this.innerBones.count = 0 + this.outerJoints.count = 0 + this.innerJoints.count = 0 + this.group.visible = false + } + + dispose() { + this.clear() + this.group.removeFromParent() + this.group.clear() + this.boneGeometry.dispose() + this.jointGeometry.dispose() + this.outerMaterial.dispose() + this.innerMaterial.dispose() + } +} + +export const createDevelopmentHandPose = ( + handedness: 'left' | 'right' +): Map => { + const side = handedness === 'left' ? -1 : 1 + const pose = new Map() + const set = (name: XRHandJoint, x: number, y: number, z: number, radius = 0.009) => { + pose.set(name, { position: new Vector3(x * side, y, z), radius }) + } + set('wrist', 0, -0.11, 0, 0.014) + set('thumb-metacarpal', 0.025, -0.055, 0.004, 0.011) + set('thumb-phalanx-proximal', 0.055, -0.025, 0.008) + set('thumb-phalanx-distal', 0.075, 0.005, 0.006) + set('thumb-tip', 0.085, 0.035, 0.004, 0.007) + const fingers = [ + ['index', 0.035, 0.105], + ['middle', 0.012, 0.125], + ['ring', -0.014, 0.116], + ['pinky', -0.038, 0.09] + ] as const + for (const [finger, x, length] of fingers) { + set(`${finger}-finger-metacarpal` as XRHandJoint, x, -0.035, 0, 0.01) + set(`${finger}-finger-phalanx-proximal` as XRHandJoint, x, 0.005, 0, 0.009) + set(`${finger}-finger-phalanx-intermediate` as XRHandJoint, x, length * 0.42, -0.004, 0.008) + set(`${finger}-finger-phalanx-distal` as XRHandJoint, x, length * 0.72, -0.008, 0.007) + set(`${finger}-finger-tip` as XRHandJoint, x, length, -0.012, 0.006) + } + return pose +} diff --git a/packages/webxr/src/immersive-scene.ts b/packages/webxr/src/immersive-scene.ts index 897238c..4ccd2f7 100644 --- a/packages/webxr/src/immersive-scene.ts +++ b/packages/webxr/src/immersive-scene.ts @@ -1,8 +1,6 @@ import { AmbientLight, - BufferGeometry, Color, - Float32BufferAttribute, GridHelper, Group, LineSegments, @@ -10,9 +8,6 @@ import { MeshBasicMaterial, PerspectiveCamera, PlaneGeometry, - Points, - PointsMaterial, - TorusGeometry, Scene, SRGBColorSpace, Timer, @@ -32,6 +27,11 @@ import { ROOM_SIZE_METERS } from './config' import { ImmersiveInteractionSystem } from './interaction-system' +import { + createDevelopmentHandPose, + HandOutlineView +} from './hand-outline-view' +import { PassthroughContrastView } from './passthrough-contrast' import { AgentLightAnimator, type RealtimeVisualActivity @@ -50,6 +50,10 @@ import { WorldStatus } from './world-status' import { StatusWindowView } from './status-window-view' +import { + resolveStatusWindowAnchorPosition, + StatusWindowAnchorTracker +} from './status-window-placement' import type { StatusActionId, StatusWindowInvocation, @@ -86,10 +90,54 @@ const viewerDirection = new Vector3() const worldCenter = new Vector3(0, 1.65, 0) const worldForward = new Vector3(0, 0, -1) const floorCenter = new Vector3() +export const ROOM_FLOOR_RENDER_ORDER = -100 +export const ROOM_GRID_RENDER_ORDER = -99 +export const ROOM_WALL_RENDER_ORDER = -101 + +export const configureRoomSurfaceRendering = ( + floor: Mesh, + grid: GridHelper, + walls: readonly Mesh[] = [] +) => { + const floorMaterials = Array.isArray(floor.material) + ? floor.material + : [floor.material] + const gridMaterials = Array.isArray(grid.material) + ? grid.material + : [grid.material] + const wallMaterials = walls.flatMap(wall => ( + Array.isArray(wall.material) ? wall.material : [wall.material] + )) + for (const material of [ + ...floorMaterials, + ...gridMaterials, + ...wallMaterials + ]) { + material.depthWrite = false + } + floor.renderOrder = ROOM_FLOOR_RENDER_ORDER + grid.renderOrder = ROOM_GRID_RENDER_ORDER + for (const wall of walls) { + wall.renderOrder = ROOM_WALL_RENDER_ORDER + } +} + +export const resolveWorldControlTargets = ( + voiceToggleEnabled: boolean, + voiceTarget: Mesh, + controlTargets: readonly Mesh[] +) => voiceToggleEnabled + ? [voiceTarget, ...controlTargets] + : [...controlTargets] + +export const shouldShowHandOutlines = ( + mode: ImmersiveSessionMode, + environmentBlendMode: XREnvironmentBlendMode +) => !(mode === 'immersive-ar' && environmentBlendMode !== 'opaque') const statusAnchorPosition = new Vector3() const menuWorldPosition = new Vector3() const menuOffset = new Vector3(0.52, -0.32, -1.15) -const fallbackStatusOffset = new Vector3(-0.18, 0, -1.12) +const fallbackStatusOffset = new Vector3(-0.08, 0, -0.4) export class ImmersiveScene { readonly scene = new Scene() @@ -118,7 +166,9 @@ export class ImmersiveScene { private readonly statusWindow = new StatusWindowView() - private readonly contrast = new Group() + private readonly contrast = new PassthroughContrastView() + + private readonly developmentHands: HandOutlineView[] = [] private readonly panels = new Map() @@ -148,6 +198,8 @@ export class ImmersiveScene { private statusInvocation: StatusWindowInvocation | null = null + private readonly statusAnchorTracker = new StatusWindowAnchorTracker() + private panelInteractionPreview: { panelId: string state: 'idle' | 'active' | 'hover' | 'grab' @@ -155,6 +207,8 @@ export class ImmersiveScene { private panelHandControlsPreview = false + private voiceToggleEnabled = true + constructor(private readonly options: ImmersiveSceneOptions) { this.renderer = new WebGLRenderer({ canvas: options.canvas, @@ -178,23 +232,24 @@ export class ImmersiveScene { this.transcriptView.group, this.controls.group, this.status.group, - this.contrast + this.contrast.group ) this.scene.add(this.statusWindow.group, this.statusWindow.menuGroup) - this.createContrastTreatments() this.setWorldCenter(new Vector3(0, 1.65, 0)) this.interaction = new ImmersiveInteractionSystem({ renderer: this.renderer, root: this.scene, getPanels: () => this.panels, - getControlTargets: () => [ + getControlTargets: () => resolveWorldControlTargets( + this.voiceToggleEnabled, this.agentLight.hitTarget, - ...this.controls.hitTargets - ], + this.controls.hitTargets + ), getStatusTargets: () => this.statusWindow.actionHits, getStatusMenuTarget: () => this.statusWindow.menuHit, isStatusOpen: () => this.statusWindow.isOpen, + isStatusFullyOpen: () => this.statusWindow.isFullyOpen, getStatusInvocation: () => this.statusInvocation, onScroll: options.onPanelScroll, onPanelInteracted: options.onPanelInteracted, @@ -208,6 +263,9 @@ export class ImmersiveScene { this.closeStatusWindow() options.onStatusAction(action) }, + onStatusPressedChanged: action => { + this.statusWindow.setPressedAction(action) + }, onInputCapabilitiesChanged: ({ fallbackMenu }) => { this.statusWindow.setMenuVisible(fallbackMenu) options.onStatusFallbackChanged(fallbackMenu) @@ -219,53 +277,6 @@ export class ImmersiveScene { this.resize() } - private createContrastTreatments() { - const ditherPositions: number[] = [] - const pointCount = 420 - const goldenAngle = Math.PI * (3 - Math.sqrt(5)) - for (let index = 0; index < pointCount; index += 1) { - const y = 1 - ((index / (pointCount - 1)) * 2) - const radius = Math.sqrt(1 - y * y) - const theta = goldenAngle * index - ditherPositions.push( - Math.cos(theta) * radius * 0.34, - y * 0.34, - Math.sin(theta) * radius * 0.34 - ) - } - const ditherGeometry = new BufferGeometry() - ditherGeometry.setAttribute( - 'position', - new Float32BufferAttribute(ditherPositions, 3) - ) - const dither = new Points( - ditherGeometry, - new PointsMaterial({ - color: '#071008', - size: 0.012, - sizeAttenuation: true, - transparent: true, - opacity: 0.38, - depthWrite: false - }) - ) - dither.name = 'alpha-passthrough-dither-shell' - dither.userData.contrast = 'dither' - const additive = new Mesh( - new TorusGeometry(0.28, 0.025, 10, 48), - new MeshBasicMaterial({ - color: '#ff6bd6', - transparent: true, - opacity: 0.88, - depthWrite: false - }) - ) - additive.name = 'additive-passthrough-shape-outline' - additive.userData.contrast = 'additive-shape' - this.contrast.add(dither, additive) - this.contrast.visible = false - } - private createRoom() { this.world.add(this.room) const floorMaterial = new MeshBasicMaterial({ @@ -311,6 +322,7 @@ export class ImmersiveScene { const right = new Mesh(wallGeometry.clone(), wallMaterial.clone()) right.position.set(ROOM_SIZE_METERS / 2, 1.8, 0) right.rotation.y = -Math.PI / 2 + configureRoomSurfaceRendering(floor, grid, [back, front, left, right]) this.room.add(back, front, left, right) } @@ -346,7 +358,7 @@ export class ImmersiveScene { ) floorCenter.set(0, 0, 0) this.room.position.copy(floorCenter) - this.contrast.position.copy(this.agentLight.group.position) + this.contrast.group.position.copy(this.agentLight.group.position) } private placeFromInitialViewer() { @@ -378,6 +390,11 @@ export class ImmersiveScene { this.activity = activity } + setVoiceToggleEnabled(enabled: boolean) { + this.voiceToggleEnabled = enabled + this.agentLight.hitTarget.visible = enabled + } + prepareAgentAwakening() { this.lightAnimator.enterDormant() } @@ -413,6 +430,7 @@ export class ImmersiveScene { private toggleStatusWindow(invocation: StatusWindowInvocation) { if (this.statusWindow.toggle(performance.now())) { if (this.statusWindow.isOpen) { + this.statusAnchorTracker.reset() this.statusInvocation = invocation this.options.onStatusOpened() } else { @@ -434,6 +452,7 @@ export class ImmersiveScene { private closeStatusWindow() { if (this.statusWindow.close(performance.now())) { + this.statusAnchorTracker.reset() this.statusInvocation = null this.options.onStatusClosed() } @@ -470,18 +489,16 @@ export class ImmersiveScene { this.environmentBlendMode = environmentBlendMode const passthrough = mode === 'immersive-ar' && environmentBlendMode !== 'opaque' + this.interaction.setHandOutlinesEnabled( + shouldShowHandOutlines(mode, environmentBlendMode) + ) this.room.visible = !passthrough this.controls.group.visible = !passthrough this.scene.background = passthrough ? null : new Color('#01040a') this.renderer.setClearColor(0x000000, passthrough ? 0 : 1) - this.contrast.visible = passthrough - for (const child of this.contrast.children) { - child.visible = environmentBlendMode === 'alpha-blend' - ? child.userData.contrast === 'dither' - : environmentBlendMode === 'additive' - ? child.userData.contrast === 'additive-shape' - : false - } + this.contrast.setBlendMode( + passthrough ? environmentBlendMode : 'opaque' + ) } private syncPanelViews() { @@ -523,7 +540,7 @@ export class ImmersiveScene { view.group.visible = true view.update(snapshot) view.placeInSlot( - snapshot.userMoved && snapshot.position + snapshot.position ? snapshot.position : placement.position, layoutNow @@ -565,6 +582,29 @@ export class ImmersiveScene { this.applyPanelInteractionPreview() } + setHandOutlinePreview(visible: boolean) { + if (!visible) { + for (const hand of this.developmentHands) { + hand.group.visible = false + } + return + } + if (this.developmentHands.length === 0) { + for (const [index, handedness] of (['left', 'right'] as const).entries()) { + const hand = new HandOutlineView(handedness) + hand.update(createDevelopmentHandPose(handedness)) + hand.group.position.set(index === 0 ? -0.52 : 0.52, 1.28, 0.22) + hand.group.rotation.set(-0.35, index === 0 ? -0.18 : 0.18, 0) + this.world.add(hand.group) + this.developmentHands.push(hand) + } + return + } + for (const hand of this.developmentHands) { + hand.group.visible = true + } + } + resize() { const width = Math.max(1, this.options.canvas.clientWidth || window.innerWidth) const height = Math.max(1, this.options.canvas.clientHeight || window.innerHeight) @@ -630,10 +670,41 @@ export class ImmersiveScene { camera.getWorldPosition(viewerPosition) const now = performance.now() const anchor = this.interaction.statusAnchor() - if (anchor && this.statusWindow.group.visible) { + if ( + this.statusInvocation === 'hand' + && this.statusWindow.group.visible + ) { + const trackedWrist = anchor + ? anchor.getWorldPosition(statusAnchorPosition) + : null + const trackedPosition = this.statusAnchorTracker.update({ + wristPosition: trackedWrist, + selectionEngaged: this.interaction.isStatusHandEngaged(), + deltaSeconds + }) + if (trackedPosition) { + this.statusWindow.group.position.copy(trackedPosition) + } else { + this.statusWindow.group.position.copy(fallbackStatusOffset) + .applyQuaternion(camera.quaternion) + .add(viewerPosition) + } + const statusTarget = viewerFacingQuaternion( + this.statusWindow.group.position, + viewerPosition + ) + smoothViewerFacingQuaternion( + this.statusWindow.group.quaternion, + statusTarget, + deltaSeconds + ) + } else if (anchor && this.statusWindow.group.visible) { anchor.getWorldPosition(statusAnchorPosition) - this.statusWindow.group.position.copy(statusAnchorPosition) - this.statusWindow.group.position.y += 0.34 + resolveStatusWindowAnchorPosition( + statusAnchorPosition, + viewerPosition, + this.statusWindow.group.position + ) const statusTarget = viewerFacingQuaternion( this.statusWindow.group.position, viewerPosition @@ -722,11 +793,15 @@ export class ImmersiveScene { this.status.dispose() this.statusWindow.dispose() this.agentLight.dispose() + this.contrast.dispose() + for (const hand of this.developmentHands) { + hand.dispose() + } + this.developmentHands.length = 0 this.scene.traverse((object) => { if ( object instanceof Mesh || object instanceof LineSegments - || object instanceof Points ) { object.geometry.dispose() if (Array.isArray(object.material)) { diff --git a/packages/webxr/src/interaction-system.ts b/packages/webxr/src/interaction-system.ts index 16fa09b..017aa55 100644 --- a/packages/webxr/src/interaction-system.ts +++ b/packages/webxr/src/interaction-system.ts @@ -1,16 +1,17 @@ import { Box3, + BoxGeometry, BufferGeometry, Line, LineBasicMaterial, Matrix4, + MathUtils, Mesh, MeshBasicMaterial, Plane, Quaternion, Raycaster, type Ray, - Sphere, SphereGeometry, Vector3, type Object3D, @@ -23,6 +24,10 @@ import { PanelInteractionModel, type PanelHit } from './panel-interaction' +import { + HandOutlineView, + resolveHandJointRadius +} from './hand-outline-view' import type { SpatialPanelView } from './panel-view' import { classifyPaneGrabIntent, @@ -30,6 +35,7 @@ import { PANE_GRAB_CLASSIFICATION_THRESHOLD_METERS, PreferredPaneInputModel, resolveDepthLockedPanePosition, + resolveHandPinchPanePosition, resolveHandScrollRate, resolvePaneDepthActivationOffset, resolvePrimaryThumbstickY, @@ -41,6 +47,7 @@ import { canActivateStatusAction, mappedMenuButtonIndex, shouldShowStatusFallbackMenu, + StatusActionInteractionModel, StatusControllerArmModel, StatusGestureModel, type StatusActionId, @@ -54,14 +61,14 @@ type SourceRuntime = { targetRay: XRTargetRaySpace grip: XRGripSpace hand: XRHandSpace + handOutline: HandOutlineView ray: Line gripMarker: Mesh inputSource: XRInputSource | null selecting: boolean pinching: boolean + statusPressActive: boolean grabbedBy: 'select' | 'squeeze' | 'pinch' | 'touch' | null - grabOffset: Vector3 - grabSphere: Sphere grabInitialPosition: Vector3 grabInitialWorldPosition: Vector3 grabInitialSourcePosition: Vector3 @@ -96,6 +103,7 @@ export type InteractionSystemOptions = { getStatusTargets: () => readonly Mesh[] getStatusMenuTarget: () => Mesh | null isStatusOpen: () => boolean + isStatusFullyOpen?: () => boolean getStatusInvocation: () => StatusWindowInvocation | null onScroll: ( panelId: string, @@ -110,6 +118,7 @@ export type InteractionSystemOptions = { onStatusToggle: (invocation: StatusWindowInvocation) => void onStatusDismiss: () => void onStatusAction: (action: StatusActionId) => void + onStatusPressedChanged?: (action: StatusActionId | null) => void onInputCapabilitiesChanged: (input: { controller: boolean hand: boolean @@ -132,16 +141,23 @@ const panelPlaneQuaternion = new Quaternion() const jointQuaternion = new Quaternion() const handBackNormal = new Vector3() const wristToViewer = new Vector3() +const viewerToWrist = new Vector3() +const viewerForward = new Vector3() const contactBounds = new Box3() const sourceDisplacement = new Vector3() const viewerLocalDisplacement = new Vector3() const panelWorldPosition = new Vector3() const contactDelta = new Vector3() const viewerQuaternion = new Quaternion() +const statusLocalPoint = new Vector3() +const statusClosestLocalPoint = new Vector3() +const statusClosestWorldPoint = new Vector3() +const statusTargetPosition = new Vector3() const PANEL_GRAB_TAP_MAX_DISTANCE_METERS = ( PANE_GRAB_CLASSIFICATION_THRESHOLD_METERS * 0.75 ) const PANEL_FOCUSED_DISTANCE_METERS = 1.8 +const STATUS_HAND_ENGAGEMENT_DISTANCE_METERS = 0.18 export const isPanelGrabTap = ( initialPosition: Vector3, @@ -149,14 +165,26 @@ export const isPanelGrabTap = ( ) => initialPosition.distanceTo(currentPosition) <= PANEL_GRAB_TAP_MAX_DISTANCE_METERS -export const resolveRayGrabPosition = ( - ray: Ray, - sphere: Sphere, - offset: Vector3, - target = new Vector3() +export const statusTargetIntersectsSphere = ( + target: Mesh, + center: Vector3, + radius: number ) => { - const intersection = ray.intersectSphere(sphere, target) - return intersection?.add(offset) ?? null + target.updateWorldMatrix(true, false) + target.worldToLocal(statusLocalPoint.copy(center)) + const width = target.geometry.parameters.width as number + const height = target.geometry.parameters.height as number + const depth = target.geometry.parameters.depth as number + statusClosestLocalPoint.set( + MathUtils.clamp(statusLocalPoint.x, -width / 2, width / 2), + MathUtils.clamp(statusLocalPoint.y, -height / 2, height / 2), + MathUtils.clamp(statusLocalPoint.z, -depth / 2, depth / 2) + ) + target.localToWorld( + statusClosestWorldPoint.copy(statusClosestLocalPoint) + ) + return statusClosestWorldPoint.distanceToSquared(center) + <= Math.max(0, radius) ** 2 } export const resolveTrackedHandJoint = ( @@ -271,12 +299,20 @@ export class ImmersiveInteractionSystem { private readonly statusGesture = new StatusGestureModel() + private readonly statusActions = new StatusActionInteractionModel() + private readonly statusControllerArm = new StatusControllerArmModel() private readonly preferredInput = new PreferredPaneInputModel() + private statusHandEngaged = false + + private statusPressedAction: StatusActionId | null = null + private lastInputCapabilities = '' + private handOutlinesEnabled = true + private disposed = false constructor(private readonly options: InteractionSystemOptions) { @@ -290,6 +326,7 @@ export class ImmersiveInteractionSystem { const targetRay = this.options.renderer.xr.getController(index) const grip = this.options.renderer.xr.getControllerGrip(index) const hand = this.options.renderer.xr.getHand(index) + const handOutline = new HandOutlineView() const ray = new Line( new BufferGeometry().setFromPoints([ @@ -316,6 +353,7 @@ export class ImmersiveInteractionSystem { ) gripMarker.name = 'generic-controller-grip' grip.add(gripMarker) + hand.add(handOutline.group) this.options.root.add(targetRay, grip, hand) const runtime: SourceRuntime = { @@ -323,14 +361,14 @@ export class ImmersiveInteractionSystem { targetRay, grip, hand, + handOutline, ray, gripMarker, inputSource: null, selecting: false, pinching: false, + statusPressActive: false, grabbedBy: null, - grabOffset: new Vector3(), - grabSphere: new Sphere(), grabInitialPosition: new Vector3(), grabInitialWorldPosition: new Vector3(), grabInitialSourcePosition: new Vector3(), @@ -355,6 +393,11 @@ export class ImmersiveInteractionSystem { runtime.inputSource = ( event as unknown as { data: XRInputSource } ).data + runtime.handOutline.setHandedness(runtime.inputSource.handedness) + runtime.statusPressActive = false + runtime.contactActionId = null + this.statusActions.updatePress(id, false) + this.statusActions.updateContact(id, null) this.preferredInput.connect({ id, handedness: runtime.inputSource.handedness, @@ -364,17 +407,30 @@ export class ImmersiveInteractionSystem { disconnected: () => { runtime.selecting = false runtime.pinching = false + runtime.statusPressActive = false this.finalizeGrab(runtime, { refresh: false }) runtime.inputSource = null + runtime.contactActionId = null + runtime.handOutline.clear() + this.statusActions.loseSource(id) this.stopHandScroll(runtime) this.preferredInput.lose(id) this.model.sourceLost(id) this.refreshPanelInteraction() }, selectstart: () => { - this.handleSelectStart(runtime, performance.now(), true) + runtime.statusPressActive = true + const freshStatusPress = this.statusActions.updatePress(id, true) + this.handleSelectStart( + runtime, + performance.now(), + true, + freshStatusPress + ) }, selectend: () => { + runtime.statusPressActive = false + this.statusActions.updatePress(id, false) runtime.selecting = false this.model.selectEnd(id) if (runtime.grabbedBy === 'select') { @@ -466,7 +522,8 @@ export class ImmersiveInteractionSystem { private handleSelectStart( runtime: SourceRuntime, now: number, - native: boolean + native: boolean, + freshStatusPress = false ) { const intersection = this.raycast(runtime) if (intersection?.object.userData.statusMenu === true) { @@ -479,11 +536,15 @@ export class ImmersiveInteractionSystem { runtime, statusAction as StatusActionId, native ? 'ray' : 'pinch', - intersection!.object + intersection!.object, + freshStatusPress ) return } - if (native && this.activateControllerContact(runtime)) { + if ( + native + && this.activateControllerContact(runtime, freshStatusPress) + ) { return } const action = intersection?.object.userData.action @@ -516,6 +577,17 @@ export class ImmersiveInteractionSystem { this.options.onPanelInteracted(hit.panelId) } runtime.selecting = true + if (hit?.zone === 'scroll-up' || hit?.zone === 'scroll-down') { + const panel = this.options.getPanels().get(hit.panelId) + if (panel) { + this.options.onScroll( + hit.panelId, + hit.zone === 'scroll-up' ? -1 : 1, + panel.maximumScrollStart + ) + } + return + } if (hit?.zone === 'move') { this.handleGrabStart(runtime, native ? 'select' : 'pinch') } @@ -538,8 +610,15 @@ export class ImmersiveInteractionSystem { runtime: SourceRuntime, action: StatusActionId, method: StatusActivation['method'], - target: Object3D + target: Object3D, + freshTransition: boolean ) { + if ( + !freshTransition + || runtime.inputSource?.handedness !== 'right' + ) { + return false + } if (target.userData.statusActionAvailable !== true) { return false } @@ -570,7 +649,26 @@ export class ImmersiveInteractionSystem { return nearest?.target ?? null } - private activateControllerContact(runtime: SourceRuntime) { + private nearestHandStatusContact(point: Vector3, radius: number) { + let nearest: { target: Mesh, distance: number } | null = null + for (const candidate of this.options.getStatusTargets()) { + const target = candidate as Mesh + if (!statusTargetIntersectsSphere(target, point, radius)) { + continue + } + target.getWorldPosition(statusTargetPosition) + const distance = statusTargetPosition.distanceToSquared(point) + if (!nearest || distance < nearest.distance) { + nearest = { target, distance } + } + } + return nearest?.target ?? null + } + + private activateControllerContact( + runtime: SourceRuntime, + freshStatusPress: boolean + ) { if ( !this.options.isStatusOpen() || runtime.inputSource?.hand @@ -582,13 +680,20 @@ export class ImmersiveInteractionSystem { const target = this.nearestStatusContact(sourcePosition, 0.055) const action = target?.userData.statusActionId return typeof action === 'string' - ? this.activateStatusAction(runtime, action as StatusActionId, 'contact', target!) + ? this.activateStatusAction( + runtime, + action as StatusActionId, + 'contact', + target!, + freshStatusPress + ) : false } private updateHandStatusContact(runtime: SourceRuntime) { if (!runtime.inputSource?.hand || !this.options.isStatusOpen()) { runtime.contactActionId = null + this.statusActions.updateContact(runtime.id, null) return } const index = resolveTrackedHandJoint( @@ -597,15 +702,39 @@ export class ImmersiveInteractionSystem { ) if (!index) { runtime.contactActionId = null + this.statusActions.updateContact(runtime.id, null) return } index.getWorldPosition(indexPosition) - const target = this.nearestStatusContact(indexPosition, 0.018) + const radius = resolveHandJointRadius(index.jointRadius) + const target = this.nearestHandStatusContact( + indexPosition, + radius + ) const action = typeof target?.userData.statusActionId === 'string' ? target.userData.statusActionId as StatusActionId : null - if (action && runtime.contactActionId !== action) { - this.activateStatusAction(runtime, action, 'contact', target!) + const freshAction = this.statusActions.updateContact(runtime.id, action) + const thumb = resolveTrackedHandJoint(runtime.hand, 'thumb-tip') + let pinchingNow = runtime.pinching + if (thumb) { + thumb.getWorldPosition(thumbPosition) + pinchingNow ||= thumbPosition.distanceTo(indexPosition) <= 0.026 + } + if ( + freshAction + && runtime.inputSource.handedness === 'right' + && !runtime.statusPressActive + && !pinchingNow + && !runtime.selecting + ) { + this.activateStatusAction( + runtime, + freshAction, + 'contact', + target!, + true + ) } runtime.contactActionId = action } @@ -813,6 +942,7 @@ export class ImmersiveInteractionSystem { const key = `${controller}:${hand}:${preferredHand}:${fallbackMenu}` for (const panel of this.options.getPanels().values()) { panel.setHandControlsVisible(preferredHand) + panel.setRayControlsVisible(controller) } if (key !== this.lastInputCapabilities) { this.lastInputCapabilities = key @@ -824,6 +954,34 @@ export class ImmersiveInteractionSystem { } } + private updateHandOutlines() { + for (const runtime of this.sources) { + const source = runtime.inputSource + if (!source?.hand) { + runtime.handOutline.clear() + continue + } + const controllerActive = this.sources.some(candidate => + candidate.inputSource?.handedness === source.handedness + && !candidate.inputSource.hand + && candidate.inputSource.targetRayMode === 'tracked-pointer' + ) + runtime.handOutline.updateFromHand( + runtime.hand, + this.handOutlinesEnabled && !controllerActive + ) + } + } + + setHandOutlinesEnabled(enabled: boolean) { + this.handOutlinesEnabled = enabled + if (!enabled) { + for (const runtime of this.sources) { + runtime.handOutline.clear() + } + } + } + private updateStatusGesture(now: number) { const leftController = this.sources.some(runtime => runtime.inputSource?.handedness === 'left' @@ -840,6 +998,7 @@ export class ImmersiveInteractionSystem { this.options.renderer.xr.getCamera().getWorldPosition(viewerPosition) let height = Number.NEGATIVE_INFINITY let facing = Number.NEGATIVE_INFINITY + let gaze = Number.NEGATIVE_INFINITY if (wrist) { wrist.getWorldPosition(sourcePosition) wrist.getWorldQuaternion(jointQuaternion) @@ -847,13 +1006,18 @@ export class ImmersiveInteractionSystem { wristToViewer.subVectors(viewerPosition, sourcePosition).normalize() height = sourcePosition.y - viewerPosition.y facing = handBackNormal.dot(wristToViewer) + this.options.renderer.xr.getCamera().getWorldDirection(viewerForward) + viewerToWrist.subVectors(sourcePosition, viewerPosition).normalize() + gaze = viewerForward.dot(viewerToWrist) } const event = this.statusGesture.update({ now, tracked: Boolean(wrist), controllerActive: leftController, wristHeightFromEyes: height, - handBackFacingViewer: facing + handBackFacingViewer: facing, + gazeAtHandDot: gaze, + rightHandEngaged: this.statusHandEngaged }, { open: this.options.isStatusOpen(), invocation: this.options.getStatusInvocation() @@ -892,6 +1056,18 @@ export class ImmersiveInteractionSystem { } statusAnchor() { + if (this.options.getStatusInvocation() === 'hand') { + const hand = this.sources.find(runtime => + runtime.inputSource?.handedness === 'left' + && Boolean(runtime.inputSource.hand) + ) + const wrist = hand + ? resolveTrackedHandJoint(hand.hand, 'wrist') + : null + if (wrist) { + return wrist + } + } const controller = this.sources.find(runtime => runtime.inputSource?.handedness === 'left' && !runtime.inputSource.hand @@ -908,6 +1084,48 @@ export class ImmersiveInteractionSystem { : null } + isStatusHandEngaged() { + return this.statusHandEngaged + } + + private updateStatusHandEngagement() { + this.statusHandEngaged = false + if (!this.options.isStatusOpen()) { + return + } + const rightHand = this.sources.find(runtime => + runtime.inputSource?.handedness === 'right' + && Boolean(runtime.inputSource.hand) + ) + const index = rightHand + ? resolveTrackedHandJoint(rightHand.hand, 'index-finger-tip') + : null + if (!index) { + return + } + index.getWorldPosition(indexPosition) + this.statusHandEngaged = this.nearestStatusContact( + indexPosition, + STATUS_HAND_ENGAGEMENT_DISTANCE_METERS + ) !== null + } + + private updateStatusPressedVisual() { + const pressed = this.sources.find(runtime => + runtime.inputSource?.handedness === 'right' + && Boolean(runtime.inputSource.hand) + && !runtime.statusPressActive + && !runtime.pinching + && !runtime.selecting + && runtime.contactActionId !== null + )?.contactActionId ?? null + if (pressed === this.statusPressedAction) { + return + } + this.statusPressedAction = pressed + this.options.onStatusPressedChanged?.(pressed) + } + private handleGrabStart( runtime: SourceRuntime, activation: NonNullable @@ -942,23 +1160,102 @@ export class ImmersiveInteractionSystem { runtime.grabInitialViewerPosition ).normalize() if (runtime.inputSource?.hand) { - runtime.grabSphere.center.copy(runtime.grabInitialViewerPosition) - runtime.grabSphere.radius = Math.max( - 0.25, - runtime.grabInitialViewerPosition.distanceTo(intersection!.point) - ) - panel.group.getWorldPosition(sourcePosition) - runtime.grabOffset.copy(sourcePosition).sub(intersection!.point) + if (!this.handPinchPosition(runtime, runtime.grabInitialSourcePosition)) { + runtime.grabbedBy = null + this.model.releaseGrab(runtime.id) + this.refreshPanelInteraction() + return + } this.refreshPanelInteraction() return } this.sourceAnchorPosition(runtime, sourcePosition) runtime.grabInitialSourcePosition.copy(sourcePosition) - panel.group.getWorldPosition(runtime.grabOffset) - runtime.grabOffset.sub(sourcePosition) this.refreshPanelInteraction() } + private handPinchPosition(runtime: SourceRuntime, target: Vector3) { + const thumb = resolveTrackedHandJoint(runtime.hand, 'thumb-tip') + const index = resolveTrackedHandJoint(runtime.hand, 'index-finger-tip') + if (!thumb || !index) { + return null + } + thumb.getWorldPosition(thumbPosition) + index.getWorldPosition(indexPosition) + return target.copy(thumbPosition).add(indexPosition).multiplyScalar(0.5) + } + + private updateSpatialGrab( + runtime: SourceRuntime, + panel: SpatialPanelView, + currentSourcePosition: Vector3 + ) { + sourceDisplacement.subVectors( + currentSourcePosition, + runtime.grabInitialSourcePosition + ) + viewerLocalDisplacement.copy(sourceDisplacement) + .applyQuaternion(runtime.grabViewerInverseQuaternion) + const previousIntent = runtime.grabIntent + runtime.grabIntent = classifyPaneGrabIntent( + viewerLocalDisplacement, + runtime.grabIntent + ) + const physicalDepth = sourceDisplacement.dot(runtime.grabSightLine) + if ( + previousIntent === 'neutral' + && runtime.grabIntent !== 'neutral' + ) { + runtime.grabDepthActivation = physicalDepth + if (runtime.inputSource?.hand) { + runtime.grabDepthActivationOffset.copy(sourceDisplacement) + } else if (runtime.grabIntent === 'depth') { + resolvePaneDepthActivationOffset({ + initialPanelPosition: runtime.grabInitialWorldPosition, + initialViewerPosition: runtime.grabInitialViewerPosition, + sightLine: runtime.grabSightLine, + sourceDisplacement, + target: runtime.grabDepthActivationOffset + }) + } + } + const position = runtime.inputSource?.hand + && runtime.grabIntent !== 'neutral' + ? resolveHandPinchPanePosition({ + initialPanelPosition: runtime.grabInitialWorldPosition, + initialViewerPosition: runtime.grabInitialViewerPosition, + initialSourcePosition: runtime.grabInitialSourcePosition, + sightLine: runtime.grabSightLine, + sourceDisplacement, + activationPhysicalDepth: runtime.grabDepthActivation, + activationSourceDisplacement: runtime.grabDepthActivationOffset, + accelerateDepth: runtime.grabIntent === 'depth', + target: panelWorldPosition + }) + : runtime.grabIntent === 'depth' + ? resolveDepthLockedPanePosition({ + initialPanelPosition: runtime.grabInitialWorldPosition, + initialViewerPosition: runtime.grabInitialViewerPosition, + sightLine: runtime.grabSightLine, + physicalDepth, + activationPhysicalDepth: runtime.grabDepthActivation, + activationOffset: runtime.grabDepthActivationOffset, + target: panelWorldPosition + }) + : panelWorldPosition.copy(runtime.grabInitialWorldPosition) + .add(sourceDisplacement) + worldPointToPanelLocal( + panel.group, + position, + position + ) + runtime.grabMoved ||= !isPanelGrabTap( + runtime.grabInitialPosition, + position + ) + panel.moveTo(position) + } + private sourceAnchorPosition(runtime: SourceRuntime, target: Vector3) { const anchor = runtime.inputSource?.gripSpace ? runtime.grip @@ -1030,7 +1327,7 @@ export class ImmersiveInteractionSystem { if (!runtime.pinching && distance <= 0.026) { runtime.pinching = true this.preferredInput.use(runtime.id, now) - this.handleSelectStart(runtime, now, false) + this.handleSelectStart(runtime, now, false, false) } else if (runtime.pinching && distance >= 0.038) { this.endSynthesizedPinch(runtime, true) } @@ -1135,7 +1432,17 @@ export class ImmersiveInteractionSystem { if (this.disposed) { return } + this.statusActions.updateWindow({ + now, + open: this.options.isStatusOpen(), + fullyOpen: this.options.isStatusFullyOpen?.() ?? false + }) + this.updateStatusHandEngagement() for (const runtime of this.sources) { + this.statusActions.updatePress( + runtime.id, + runtime.statusPressActive + ) const intersection = this.raycast(runtime) this.model.hover( runtime.id, @@ -1155,83 +1462,26 @@ export class ImmersiveInteractionSystem { continue } if (runtime.inputSource?.hand) { - this.updateTargetRay(runtime) - this.options.renderer.xr.getCamera() - .getWorldPosition(runtime.grabSphere.center) - const position = resolveRayGrabPosition( - this.raycaster.ray, - runtime.grabSphere, - runtime.grabOffset, + const pinchPosition = this.handPinchPosition( + runtime, sourcePosition ) - if (position) { - worldPointToPanelLocal( - panel.group, - position, - position - ) - runtime.grabMoved ||= !isPanelGrabTap( - runtime.grabInitialPosition, - position - ) - panel.moveTo(position) + if (pinchPosition) { + this.updateSpatialGrab(runtime, panel, pinchPosition) } } else { this.sourceAnchorPosition(runtime, sourcePosition) - sourceDisplacement.subVectors( - sourcePosition, - runtime.grabInitialSourcePosition - ) - viewerLocalDisplacement.copy(sourceDisplacement) - .applyQuaternion(runtime.grabViewerInverseQuaternion) - const previousIntent = runtime.grabIntent - runtime.grabIntent = classifyPaneGrabIntent( - viewerLocalDisplacement, - runtime.grabIntent - ) - const physicalDepth = sourceDisplacement.dot(runtime.grabSightLine) - if ( - previousIntent === 'neutral' - && runtime.grabIntent === 'depth' - ) { - runtime.grabDepthActivation = physicalDepth - resolvePaneDepthActivationOffset({ - initialPanelPosition: runtime.grabInitialWorldPosition, - initialViewerPosition: runtime.grabInitialViewerPosition, - sightLine: runtime.grabSightLine, - sourceDisplacement, - target: runtime.grabDepthActivationOffset - }) - } - const position = runtime.grabIntent === 'depth' - ? resolveDepthLockedPanePosition({ - initialPanelPosition: runtime.grabInitialWorldPosition, - initialViewerPosition: runtime.grabInitialViewerPosition, - sightLine: runtime.grabSightLine, - physicalDepth, - activationPhysicalDepth: runtime.grabDepthActivation, - activationOffset: runtime.grabDepthActivationOffset, - target: panelWorldPosition - }) - : panelWorldPosition.copy(runtime.grabInitialWorldPosition) - .add(sourceDisplacement) - worldPointToPanelLocal( - panel.group, - position, - position - ) - runtime.grabMoved ||= !isPanelGrabTap( - runtime.grabInitialPosition, - position - ) - panel.moveTo(position) + this.updateSpatialGrab(runtime, panel, sourcePosition) } } } } + this.updateStatusPressedVisual() + this.updateHandOutlines() this.updateInputCapabilities() this.updateStatusGesture(now) this.updateControllerArmDismissal(now) + this.statusActions.finishFrame(now) this.refreshPanelInteraction() } @@ -1239,6 +1489,7 @@ export class ImmersiveInteractionSystem { this.disposed = true this.model.clear() this.preferredInput.clear() + this.options.onStatusPressedChanged?.(null) for (const runtime of this.sources) { if (runtime.listeners) { runtime.targetRay.removeEventListener( @@ -1271,6 +1522,7 @@ export class ImmersiveInteractionSystem { runtime.ray.material.dispose() runtime.gripMarker.geometry.dispose() runtime.gripMarker.material.dispose() + runtime.handOutline.dispose() runtime.targetRay.clear() runtime.grip.clear() runtime.hand.clear() diff --git a/packages/webxr/src/main.ts b/packages/webxr/src/main.ts index 1825610..a622d1c 100644 --- a/packages/webxr/src/main.ts +++ b/packages/webxr/src/main.ts @@ -12,6 +12,7 @@ import { import type { ImmersiveScene } from './immersive-scene' import { resolveImmersiveVoiceActivity, + voiceSessionActive, VoiceRuntime } from './voice-runtime' import { coordinateRealtimeAutoStart } from './realtime-auto-start' @@ -67,6 +68,12 @@ const developmentBlendPreview = developmentKitchenSink ? searchParams.get('blend') as 'alpha-blend' | 'additive' : null const developmentStatusPreview = searchParams.get('status') !== '0' +const developmentBackground = (() => { + const background = searchParams.get('background') + return background === 'bright' || background === 'dark' + ? background + : 'textured' +})() const developmentPaneState = (() => { const state = searchParams.get('paneState') return state === 'idle' || state === 'hover' || state === 'grab' @@ -117,13 +124,6 @@ const disposeConnectedRuntimes = async () => { ].filter((task): task is Promise => Boolean(task))) } -const sessionActive = (snapshot: RealtimeConversationSnapshot) => - snapshot.state === 'requesting-permission' - || snapshot.state === 'creating-offer' - || snapshot.state === 'starting' - || snapshot.state === 'connected' - || snapshot.state === 'stopping' - const statusVoiceState = () => { if (!latestVoice) { return 'unavailable' as const @@ -131,7 +131,7 @@ const statusVoiceState = () => { if (latestVoice.autoplayBlocked) { return 'resume-audio' as const } - return sessionActive(latestVoice) ? 'active' as const : 'inactive' as const + return voiceSessionActive(latestVoice) ? 'active' as const : 'inactive' as const } const updateStatusWindow = () => { @@ -194,6 +194,7 @@ const showEntry = () => { const updateVoiceUi = (snapshot: RealtimeConversationSnapshot) => { latestVoice = snapshot + immersiveScene?.setVoiceToggleEnabled(!voiceSessionActive(snapshot)) updateStatusWindow() immersiveScene?.setTranscript(snapshot.transcripts, snapshot.generation) immersiveScene?.setActivity( @@ -236,7 +237,7 @@ const ensureScene = async () => { reducedEffects: () => reducedEffects.checked, onAction: (action) => { if (action === 'toggle-voice') { - void toggleVoice() + void activateAgentLightVoice() } else { void exitImmersive() } @@ -277,6 +278,9 @@ const ensureScene = async () => { } }) updateStatusWindow() + immersiveScene.setVoiceToggleEnabled( + !latestVoice || !voiceSessionActive(latestVoice) + ) return immersiveScene }) } @@ -303,7 +307,7 @@ const startWorkspaceRuntime = async () => { latestWorkspace = snapshot lastWorkspaceError = snapshot.error immersiveScene?.setPanels(snapshot.panels) - if (!voiceRuntime || !sessionActive(voiceRuntime.getSnapshot())) { + if (!voiceRuntime || !voiceSessionActive(voiceRuntime.getSnapshot())) { immersiveScene?.setActivity(snapshot.activity) immersiveScene?.setTranscript( snapshot.transcripts, @@ -344,7 +348,7 @@ const toggleVoice = async () => { await voiceRuntime.toggle() return } - if (sessionActive(snapshot)) { + if (voiceSessionActive(snapshot)) { await voiceRuntime.stop() immersiveScene?.prepareAgentAwakening() return @@ -362,6 +366,16 @@ const toggleVoice = async () => { } } +const activateAgentLightVoice = async () => { + if ( + voiceRuntime + && voiceSessionActive(voiceRuntime.getSnapshot()) + ) { + return + } + await toggleVoice() +} + const transitionSessionMode = async (mode: ImmersiveSessionMode) => { const previous = activeSession if (!previous || transitionTarget) { @@ -592,6 +606,7 @@ const enterDebugScene = async () => { scene.setPanels(fixture.panels) scene.setPanelInteractionPreview(developmentPaneId, developmentPaneState) scene.setPanelHandControlsPreview(developmentHandControls) + scene.setHandOutlinePreview(true) scene.setStatus( 'Kitchen sink · non-immersive texture and layout preview' ) @@ -637,9 +652,15 @@ const enterDebugScene = async () => { scene.openStatusForPreview() } if (developmentBlendPreview) { - canvas.style.background = developmentBlendPreview === 'alpha-blend' - ? 'linear-gradient(135deg, #e9e2cb, #7ea0b0)' - : 'linear-gradient(135deg, #d9c995, #446d82)' + const surrogateBackgrounds = { + bright: 'linear-gradient(135deg, #fffef4, #c7e7f3)', + dark: 'linear-gradient(135deg, #05070b, #142331)', + textured: [ + 'repeating-linear-gradient(35deg, rgba(255,255,255,0.18) 0 8px, rgba(14,32,40,0.08) 8px 17px)', + 'linear-gradient(135deg, #eadfbc, #527b8c)' + ].join(',') + } + canvas.style.background = surrogateBackgrounds[developmentBackground] scene.setSessionVisualMode('immersive-ar', developmentBlendPreview) } return diff --git a/packages/webxr/src/pane-manipulation.ts b/packages/webxr/src/pane-manipulation.ts index 4f3736d..abcfd56 100644 --- a/packages/webxr/src/pane-manipulation.ts +++ b/packages/webxr/src/pane-manipulation.ts @@ -5,6 +5,7 @@ export type PaneGrabIntent = 'neutral' | 'free' | 'depth' export const PANE_GRAB_CLASSIFICATION_THRESHOLD_METERS = 0.045 export const PANE_GRAB_DEPTH_RATIO = 1.35 export const PANE_DEPTH_ACCELERATION = 3.2 +export const PANE_HAND_PINCH_MIN_SOURCE_DISTANCE_METERS = 0.25 export const PANE_MIN_VIEWER_DISTANCE_METERS = 0.65 export const PANE_MAX_VIEWER_DISTANCE_METERS = 4.5 export const PANE_SCROLL_DEAD_ZONE = 0.22 @@ -101,6 +102,73 @@ export const resolveDepthLockedPanePosition = (input: { ) .add(input.activationOffset) +/** + * Accelerates only the sight-line component of a physical hand pull. Unlike + * controller depth lock, the live lateral fingertip displacement remains 1:1 + * so a held pinch can still place the pane left/right and up/down. + */ +export const resolveHandPinchPanePosition = (input: { + initialPanelPosition: Vector3 + initialViewerPosition: Vector3 + initialSourcePosition: Vector3 + sightLine: Vector3 + sourceDisplacement: Vector3 + activationPhysicalDepth: number + activationSourceDisplacement: Vector3 + accelerateDepth: boolean + target?: Vector3 +}) => { + const physicalDepth = input.sourceDisplacement.dot(input.sightLine) + const initialDistance = input.initialViewerPosition.distanceTo( + input.initialPanelPosition + ) + const resolvedDepth = input.accelerateDepth + ? resolveAcceleratedPaneDepth({ + initialDistance, + physicalDepth, + activationPhysicalDepth: input.activationPhysicalDepth + }) + : initialDistance + physicalDepth + const currentSourceDistance = Math.hypot( + input.initialSourcePosition.x + input.sourceDisplacement.x + - input.initialViewerPosition.x, + input.initialSourcePosition.y + input.sourceDisplacement.y + - input.initialViewerPosition.y, + input.initialSourcePosition.z + input.sourceDisplacement.z + - input.initialViewerPosition.z + ) + const lateralScale = Math.max( + 1, + resolvedDepth / Math.max( + currentSourceDistance, + PANE_HAND_PINCH_MIN_SOURCE_DISTANCE_METERS + ) + ) + const activationLateralDepth = input.activationSourceDisplacement.dot( + input.sightLine + ) + return (input.target ?? new Vector3()) + .copy(input.initialViewerPosition) + .addScaledVector( + input.sightLine, + resolvedDepth + ) + .add(input.activationSourceDisplacement) + .addScaledVector(input.sightLine, -activationLateralDepth) + .addScaledVector( + input.sourceDisplacement, + lateralScale + ) + .addScaledVector( + input.activationSourceDisplacement, + -lateralScale + ) + .addScaledVector( + input.sightLine, + -(physicalDepth - activationLateralDepth) * lateralScale + ) +} + export const applyThumbstickDeadZone = ( value: number, deadZone = PANE_SCROLL_DEAD_ZONE diff --git a/packages/webxr/src/panel-layout.ts b/packages/webxr/src/panel-layout.ts index ad56a82..fe75bd1 100644 --- a/packages/webxr/src/panel-layout.ts +++ b/packages/webxr/src/panel-layout.ts @@ -20,6 +20,10 @@ export type NewPanelSlotAssignment = { id: string slot: number } | null + overflowed: { + id: string + } | null + position: SpatialPoint | null } const slots: SpatialPoint[] = [ @@ -34,6 +38,53 @@ const slots: SpatialPoint[] = [ ] const frontSlots = [0, 1, 2, 3] as const +const PANEL_PROJECTED_WIDTH_METERS = 1.55 +const PANEL_PROJECTED_HEIGHT_METERS = 0.92 +const PANEL_FORWARD_LAYER_GAP_METERS = 0.28 +const PANEL_MAX_AUTOMATIC_FORWARD_Z = 1.7 + +const projectedManualOverlaps = ( + frontSlot: number, + panels: readonly SpatialPanelSnapshot[] +) => { + const anchor = slots[frontSlot]! + return panels.filter(panel => + panel.userMoved + && panel.position !== null + && Math.abs(panel.position.x - anchor.x) < PANEL_PROJECTED_WIDTH_METERS + && Math.abs(panel.position.y - anchor.y) < PANEL_PROJECTED_HEIGHT_METERS + ) +} + +const rankedFrontSlots = ( + panels: readonly SpatialPanelSnapshot[] +) => [...frontSlots].sort((first, second) => ( + projectedManualOverlaps(first, panels).length + - projectedManualOverlaps(second, panels).length +) || first - second) + +const resolveNewPanelPosition = ( + frontSlot: number, + panels: readonly SpatialPanelSnapshot[] +) => { + const overlapping = projectedManualOverlaps(frontSlot, panels) + if (overlapping.length === 0) { + return null + } + const anchor = slots[frontSlot]! + return { + ...anchor, + z: Math.min( + PANEL_MAX_AUTOMATIC_FORWARD_Z, + Math.max( + anchor.z, + ...overlapping.map(panel => ( + panel.position!.z + PANEL_FORWARD_LAYER_GAP_METERS + )) + ) + ) + } +} export const assignNewPanelToFrontSlot = ( panels: readonly SpatialPanelSnapshot[] @@ -48,41 +99,53 @@ export const assignNewPanelToFrontSlot = ( ) .map(panel => [panel.slot!, panel]) ) - const emptyAnchor = frontSlots.find(frontSlot => + const ranked = rankedFrontSlots(panels) + const result = ( + slot: number, + displaced: NewPanelSlotAssignment['displaced'] = null, + overflowed: NewPanelSlotAssignment['overflowed'] = null + ): NewPanelSlotAssignment => ({ + slot, + displaced, + overflowed, + position: resolveNewPanelPosition(slot, panels) + }) + const emptyAnchor = ranked.find(frontSlot => !occupantBySlot.has(frontSlot) && !occupantBySlot.has(frontSlot + frontSlots.length) ) if (emptyAnchor != null) { - return { - slot: emptyAnchor, - displaced: null - } + return result(emptyAnchor) } - const openFront = frontSlots.find(frontSlot => + const openFront = ranked.find(frontSlot => !occupantBySlot.has(frontSlot) ) if (openFront != null) { - return { - slot: openFront, - displaced: null - } + return result(openFront) } - for (const frontSlot of frontSlots) { + for (const frontSlot of ranked) { const backSlot = frontSlot + frontSlots.length const occupant = occupantBySlot.get(frontSlot) if (occupant && !occupantBySlot.has(backSlot)) { - return { - slot: frontSlot, - displaced: { - id: occupant.id, - slot: backSlot - } - } + return result(frontSlot, { + id: occupant.id, + slot: backSlot + }) } } - return null + + const frontSlot = ranked[0]! + const backSlot = frontSlot + frontSlots.length + const frontOccupant = occupantBySlot.get(frontSlot)! + const backOccupant = occupantBySlot.get(backSlot)! + return result(frontSlot, { + id: frontOccupant.id, + slot: backSlot + }, { + id: backOccupant.id + }) } export const allocatePanelSlots = ( @@ -90,12 +153,12 @@ export const allocatePanelSlots = ( center: SpatialPoint ): PanelPlacement[] => { const stable = [...panels].sort((first, second) => { - if (first.background !== second.background) { - return first.background ? -1 : 1 - } if (first.slot != null !== (second.slot != null)) { return first.slot != null ? -1 : 1 } + if (first.background !== second.background) { + return first.background ? -1 : 1 + } return first.id.localeCompare(second.id) }) const occupied = new Set() @@ -122,6 +185,17 @@ export const allocatePanelSlots = ( continue } + if (panel.position && panel.slot != null && !occupied.has(panel.slot)) { + occupied.add(panel.slot) + placements.push({ + id: panel.id, + slot: panel.slot, + position: { ...panel.position }, + overflow: false + }) + continue + } + let slot = panel.slot if (slot == null || slot < 0 || slot >= slots.length || occupied.has(slot)) { slot = slots.findIndex((_candidate, candidateIndex) => !occupied.has(candidateIndex)) diff --git a/packages/webxr/src/panel-model.ts b/packages/webxr/src/panel-model.ts index e784602..2ff1fb5 100644 --- a/packages/webxr/src/panel-model.ts +++ b/packages/webxr/src/panel-model.ts @@ -130,12 +130,23 @@ export class SpatialPanelModel { const existing = this.panels.get(input.id) if (!existing) { const assignment = assignNewPanelToFrontSlot(this.snapshots()) + if (assignment?.overflowed) { + const overflowed = this.panels.get(assignment.overflowed.id) + if (overflowed) { + this.panels.set(overflowed.id, { + ...overflowed, + slot: null, + position: null + }) + } + } if (assignment?.displaced) { const displaced = this.panels.get(assignment.displaced.id) if (displaced) { this.panels.set(displaced.id, { ...displaced, - slot: assignment.displaced.slot + slot: assignment.displaced.slot, + position: null }) } } @@ -148,7 +159,7 @@ export class SpatialPanelModel { scrollOffset: Number.POSITIVE_INFINITY, autoFollow: true, userMoved: false, - position: null, + position: assignment?.position ?? null, slot: assignment?.slot ?? null, fileTransitionStartedAt: now } diff --git a/packages/webxr/src/panel-view.ts b/packages/webxr/src/panel-view.ts index fd9644e..bac894f 100644 --- a/packages/webxr/src/panel-view.ts +++ b/packages/webxr/src/panel-view.ts @@ -120,6 +120,18 @@ export const resolvePanelInteractionLayout = ( } } +export const resolvePanelOverflowControlLayout = ( + width: number, + height: number +) => { + const content = resolvePanelInteractionLayout(width, height).content + const inset = 0.04 + return { + upY: content.y + (content.height / 2) - inset, + downY: content.y - (content.height / 2) + inset + } +} + export const createPanelContentRenderSignature = (input: { title: string status: string @@ -358,6 +370,8 @@ export class SpatialPanelView { private hasContentBelow = false + private rayControlsVisible = false + get maximumScrollStart() { const metrics = this.surface.metrics return Math.max(0, metrics.totalLineCount - metrics.visibleLineCount) @@ -469,15 +483,15 @@ export class SpatialPanelView { this.overflowUp.position.z = 0.05 this.overflowDown.position.z = 0.05 this.scrollUpHit = new Mesh( - new BoxGeometry(0.13, 0.075, 0.07), + new BoxGeometry(0.22, 0.13, 0.1), invisibleMaterial.clone() ) this.scrollDownHit = new Mesh( - new BoxGeometry(0.13, 0.075, 0.07), + new BoxGeometry(0.22, 0.13, 0.1), invisibleMaterial.clone() ) - this.scrollUpHit.position.z = 0.045 - this.scrollDownHit.position.z = 0.045 + this.scrollUpHit.position.z = 0.015 + this.scrollDownHit.position.z = 0.015 this.scrollUpHit.userData = { panelId: snapshot.id, hitZone: 'scroll-up' @@ -677,16 +691,18 @@ export class SpatialPanelView { } private positionOverflowControls() { - const inset = 0.055 - this.overflowUp.position.set(0, (this.height / 2) - inset, 0.045) - this.overflowDown.position.set(0, (-this.height / 2) + inset, 0.045) + const layout = resolvePanelOverflowControlLayout(this.width, this.height) + this.overflowUp.position.set(0, layout.upY, 0.045) + this.overflowDown.position.set(0, layout.downY, 0.045) } private updateOverflowVisibility() { - this.overflowUp.visible = this.handControlsVisible && this.hasContentAbove + const scrollControlsVisible = this.handControlsVisible + || this.rayControlsVisible + this.overflowUp.visible = scrollControlsVisible && this.hasContentAbove this.overflowDown.visible = this.hasContentBelow - this.scrollDownHit.visible = this.handControlsVisible && this.hasContentBelow - this.scrollUpHit.visible = this.handControlsVisible && this.hasContentAbove + this.scrollDownHit.visible = scrollControlsVisible && this.hasContentBelow + this.scrollUpHit.visible = scrollControlsVisible && this.hasContentAbove } setHandControlsVisible(visible: boolean) { @@ -697,6 +713,14 @@ export class SpatialPanelView { this.updateOverflowVisibility() } + setRayControlsVisible(visible: boolean) { + if (this.rayControlsVisible === visible) { + return + } + this.rayControlsVisible = visible + this.updateOverflowVisibility() + } + updateAnimation(now: number, reducedEffects = false) { this.animationNow = now if (this.layoutAnimating) { diff --git a/packages/webxr/src/passthrough-contrast.ts b/packages/webxr/src/passthrough-contrast.ts new file mode 100644 index 0000000..0d57ab0 --- /dev/null +++ b/packages/webxr/src/passthrough-contrast.ts @@ -0,0 +1,93 @@ +import { + CanvasTexture, + Group, + Mesh, + MeshBasicMaterial, + NormalBlending, + Sprite, + SpriteMaterial, + TorusGeometry +} from 'three' + +export const PASSTHROUGH_DARK_FEATHER_STOPS = [ + [0, 'rgba(5, 7, 9, 0.16)'], + [0.34, 'rgba(5, 7, 9, 0.14)'], + [0.54, 'rgba(5, 7, 9, 0.09)'], + [0.74, 'rgba(5, 7, 9, 0.04)'], + [1, 'rgba(5, 7, 9, 0)'] +] as const + +const createDarkFeatherTexture = () => { + const canvas = document.createElement('canvas') + canvas.width = 256 + canvas.height = 256 + const context = canvas.getContext('2d') + if (!context) { + throw new Error('Could not create the passthrough feather texture.') + } + const gradient = context.createRadialGradient(128, 128, 0, 128, 128, 128) + for (const [offset, color] of PASSTHROUGH_DARK_FEATHER_STOPS) { + gradient.addColorStop(offset, color) + } + context.fillStyle = gradient + context.fillRect(0, 0, 256, 256) + return new CanvasTexture(canvas) +} + +export class PassthroughContrastView { + readonly group = new Group() + + readonly feather: Sprite + + readonly additiveOutline: Mesh + + private readonly featherTexture = createDarkFeatherTexture() + + private readonly featherMaterial = new SpriteMaterial({ + map: this.featherTexture, + color: '#ffffff', + transparent: true, + opacity: 0.84, + blending: NormalBlending, + depthWrite: false, + depthTest: true, + toneMapped: false + }) + + constructor() { + this.group.name = 'passthrough-orb-contrast' + this.feather = new Sprite(this.featherMaterial) + this.feather.name = 'alpha-passthrough-dark-feather' + this.feather.scale.setScalar(1.48) + this.feather.userData.contrast = 'dark-feather' + this.feather.userData.pattern = 'smooth-radial-gradient' + this.additiveOutline = new Mesh( + new TorusGeometry(0.28, 0.025, 10, 48), + new MeshBasicMaterial({ + color: '#ff6bd6', + transparent: true, + opacity: 0.88, + depthWrite: false + }) + ) + this.additiveOutline.name = 'additive-passthrough-shape-outline' + this.additiveOutline.userData.contrast = 'additive-shape' + this.group.add(this.feather, this.additiveOutline) + this.setBlendMode('opaque') + } + + setBlendMode(environmentBlendMode: XREnvironmentBlendMode) { + this.group.visible = environmentBlendMode !== 'opaque' + this.feather.visible = environmentBlendMode === 'alpha-blend' + this.additiveOutline.visible = environmentBlendMode === 'additive' + } + + dispose() { + this.featherTexture.dispose() + this.featherMaterial.dispose() + this.additiveOutline.geometry.dispose() + this.additiveOutline.material.dispose() + this.group.removeFromParent() + this.group.clear() + } +} diff --git a/packages/webxr/src/status-window-model.ts b/packages/webxr/src/status-window-model.ts index a5fd8d4..585706d 100644 --- a/packages/webxr/src/status-window-model.ts +++ b/packages/webxr/src/status-window-model.ts @@ -24,6 +24,8 @@ export type StatusAction = { id: StatusActionId label: string state: string | null + presentation: 'toggle' | 'button' + checked: boolean | null available: boolean disabledReason: string | null inputPolicy: StatusActionInputPolicy @@ -140,6 +142,8 @@ export const createStatusActions = ( id: 'passthrough', label: 'Passthrough', state: state.passthroughActive ? 'On' : 'Off', + presentation: 'toggle', + checked: state.passthroughActive, available: state.passthroughSupported, disabledReason: state.passthroughSupported ? null @@ -150,17 +154,21 @@ export const createStatusActions = ( id: 'recenter', label: 'Recenter workspace', state: null, + presentation: 'button', + checked: null, available: true, disabledReason: null, inputPolicy: 'controller-or-touch' }, { id: 'voice', - label: state.voiceState === 'resume-audio' - ? 'Resume audio' - : state.voiceState === 'active' - ? 'Stop voice' - : 'Start voice', - state: state.voiceState === 'active' ? 'On' : 'Off', + label: state.voiceState === 'resume-audio' ? 'Resume audio' : 'Voice', + state: state.voiceState === 'resume-audio' + ? null + : state.voiceState === 'active' ? 'On' : 'Off', + presentation: state.voiceState === 'resume-audio' ? 'button' : 'toggle', + checked: state.voiceState === 'resume-audio' + ? null + : state.voiceState === 'active', available: state.voiceState !== 'unavailable', disabledReason: state.voiceState === 'unavailable' ? 'Realtime voice is unavailable.' @@ -170,6 +178,8 @@ export const createStatusActions = ( id: 'reduced-effects', label: 'Reduced effects', state: state.reducedEffects ? 'On' : 'Off', + presentation: 'toggle', + checked: state.reducedEffects, available: true, disabledReason: null, inputPolicy: 'controller-or-touch' @@ -177,6 +187,8 @@ export const createStatusActions = ( id: 'exit', label: 'Exit immersive', state: null, + presentation: 'button', + checked: null, available: true, disabledReason: null, inputPolicy: 'any' @@ -213,82 +225,324 @@ export type StatusGestureSample = { controllerActive: boolean wristHeightFromEyes: number handBackFacingViewer: number + gazeAtHandDot: number + rightHandEngaged?: boolean } export type StatusWindowInvocation = 'controller' | 'hand' | 'fallback' +export type StatusWindowPhase = 'closed' | 'opening' | 'open' | 'closing' + +export type StatusActionInteractionPhase = + | 'closed' + | 'emerging' + | 'arming' + | 'armed' + +export const STATUS_ACTION_ARMING_GRACE_MS = 180 + +type StatusActionSourceState = { + pressActive: boolean + pressNeutral: boolean + contactAction: StatusActionId | null + contactNeutral: boolean +} + +const createStatusActionSourceState = (): StatusActionSourceState => ({ + pressActive: false, + pressNeutral: false, + contactAction: null, + contactNeutral: false +}) + +/** + * Keeps status-window presentation separate from activation. The gate does not + * arm until the opening animation and an additional grace period have both + * completed. At that boundary it snapshots every observed press/contact so an + * input that was already held or overlapping cannot become a fresh action. + */ +export class StatusActionInteractionModel { + private currentPhase: StatusActionInteractionPhase = 'closed' + + private armAt: number | null = null + + private readonly sources = new Map() + + get phase() { + return this.currentPhase + } + + private source(sourceId: string) { + let source = this.sources.get(sourceId) + if (!source) { + source = createStatusActionSourceState() + this.sources.set(sourceId, source) + } + return source + } + + private close() { + this.currentPhase = 'closed' + this.armAt = null + this.sources.clear() + } + + updateWindow(input: { + now: number + open: boolean + fullyOpen: boolean + }) { + if (!input.open) { + this.close() + return + } + if (this.currentPhase === 'closed') { + this.currentPhase = 'emerging' + } + if (!input.fullyOpen) { + this.currentPhase = 'emerging' + this.armAt = null + return + } + if (this.currentPhase === 'emerging') { + this.currentPhase = 'arming' + this.armAt = input.now + STATUS_ACTION_ARMING_GRACE_MS + } + } + + finishFrame(now: number) { + if ( + this.currentPhase !== 'arming' + || this.armAt == null + || now < this.armAt + ) { + return + } + this.currentPhase = 'armed' + for (const source of this.sources.values()) { + source.pressNeutral = !source.pressActive + source.contactNeutral = source.contactAction === null + } + } + + updatePress(sourceId: string, pressed: boolean) { + const source = this.source(sourceId) + const freshPress = pressed && !source.pressActive + source.pressActive = pressed + if (this.currentPhase !== 'armed') { + return false + } + if (!pressed) { + source.pressNeutral = true + return false + } + if (!freshPress || !source.pressNeutral) { + return false + } + source.pressNeutral = false + return true + } + + updateContact(sourceId: string, action: StatusActionId | null) { + const source = this.source(sourceId) + const freshContact = action !== null && source.contactAction === null + source.contactAction = action + if (this.currentPhase !== 'armed') { + return null + } + if (action === null) { + source.contactNeutral = true + return null + } + if (!freshContact || !source.contactNeutral) { + return null + } + source.contactNeutral = false + return action + } + + loseSource(sourceId: string) { + this.sources.delete(sourceId) + } +} + export const STATUS_GESTURE_THRESHOLDS = { - openHeightMeters: -0.28, - closeHeightMeters: -0.48, - openFacingDot: 0.55, - closeFacingDot: 0.25, - holdMs: 450, - lowerHoldMs: 180, - cooldownMs: 650 + openHeightMeters: -0.2, + openFacingDot: 0.68, + openGazeDot: 0.86, + closeFacingDot: -0.3, + closeFacingResetDot: 0.05, + lowerTravelMeters: 0.18, + lowerEndHeightMeters: -0.36, + maximumPoseJumpMeters: 0.08, + maximumWristSpeedMetersPerSecond: 1.6, + holdMs: 700, + turnAwayHoldMs: 560, + lowerHoldMs: 320, + trackingRecoveryMs: 180, + selectionProtectionMs: 700, + cooldownMs: 750 } as const export class StatusGestureModel { private candidateSince: number | null = null + + private turnAwaySince: number | null = null + private lowerSince: number | null = null - private cooldownUntil = 0 - private trackingLostSince: number | null = null + private openReferenceHeight: number | null = null + + private lastTrackedHeight: number | null = null + + private lastTrackedAt: number | null = null + + private trackingStableSince: number | null = null + + private selectionProtectedUntil = 0 + + private wasOpen = false + + private cooldownUntil = 0 suppress(now: number) { this.candidateSince = null + this.resetDismissalIntent() this.cooldownUntil = now + STATUS_GESTURE_THRESHOLDS.cooldownMs } + private resetDismissalIntent() { + this.turnAwaySince = null + this.lowerSince = null + } + + private resetTrackingContinuity() { + this.resetDismissalIntent() + this.lastTrackedHeight = null + this.lastTrackedAt = null + this.trackingStableSince = null + } + update( sample: StatusGestureSample, state: { open: boolean, invocation: StatusWindowInvocation | null } ) { if (sample.controllerActive) { this.candidateSince = null - this.lowerSince = null - this.trackingLostSince = null + this.resetDismissalIntent() return null } if (state.open && state.invocation !== 'hand') { this.candidateSince = null - this.lowerSince = null - this.trackingLostSince = null + this.resetDismissalIntent() return null } if (!sample.tracked) { this.candidateSince = null - this.lowerSince = null - if (!state.open || state.invocation !== 'hand') { - this.trackingLostSince = null - return null - } - this.trackingLostSince ??= sample.now - return sample.now - this.trackingLostSince >= 300 - ? 'close' as const - : null + this.resetTrackingContinuity() + return null } - this.trackingLostSince = null if (state.open) { - const lowered = sample.wristHeightFromEyes - <= STATUS_GESTURE_THRESHOLDS.closeHeightMeters - || sample.handBackFacingViewer - <= STATUS_GESTURE_THRESHOLDS.closeFacingDot - if (!lowered) { - this.lowerSince = null + if (!this.wasOpen) { + this.openReferenceHeight = sample.wristHeightFromEyes + this.resetTrackingContinuity() + } + this.wasOpen = true + this.candidateSince = null + + const previousHeight = this.lastTrackedHeight + const previousTrackedAt = this.lastTrackedAt + this.lastTrackedHeight = sample.wristHeightFromEyes + this.lastTrackedAt = sample.now + const trackingIntervalSeconds = previousTrackedAt === null + ? 0 + : Math.max(0, sample.now - previousTrackedAt) / 1_000 + const maximumContinuousMovement = ( + STATUS_GESTURE_THRESHOLDS.maximumPoseJumpMeters + + trackingIntervalSeconds + * STATUS_GESTURE_THRESHOLDS.maximumWristSpeedMetersPerSecond + ) + if ( + previousHeight !== null + && Math.abs(sample.wristHeightFromEyes - previousHeight) + > maximumContinuousMovement + ) { + // A large single-frame wrist jump is characteristic of hand + // occlusion/reacquisition, not a deliberate lowering trajectory. + this.openReferenceHeight = null + this.resetDismissalIntent() + this.trackingStableSince = sample.now + return null + } + + this.trackingStableSince ??= sample.now + this.openReferenceHeight = Math.max( + this.openReferenceHeight ?? sample.wristHeightFromEyes, + sample.wristHeightFromEyes + ) + + if (sample.rightHandEngaged) { + this.selectionProtectedUntil = sample.now + + STATUS_GESTURE_THRESHOLDS.selectionProtectionMs + this.openReferenceHeight = sample.wristHeightFromEyes + this.resetDismissalIntent() return null } - this.lowerSince ??= sample.now - if (sample.now - this.lowerSince >= STATUS_GESTURE_THRESHOLDS.lowerHoldMs) { + if ( + sample.now < this.selectionProtectedUntil + || sample.now - this.trackingStableSince + < STATUS_GESTURE_THRESHOLDS.trackingRecoveryMs + ) { + this.openReferenceHeight = sample.wristHeightFromEyes + this.resetDismissalIntent() + return null + } + + if ( + sample.handBackFacingViewer + <= STATUS_GESTURE_THRESHOLDS.closeFacingDot + ) { + this.turnAwaySince ??= sample.now + } else if ( + sample.handBackFacingViewer + >= STATUS_GESTURE_THRESHOLDS.closeFacingResetDot + ) { + this.turnAwaySince = null + } + + const lowered = ( + (this.openReferenceHeight - sample.wristHeightFromEyes) + >= STATUS_GESTURE_THRESHOLDS.lowerTravelMeters + && sample.wristHeightFromEyes + <= STATUS_GESTURE_THRESHOLDS.lowerEndHeightMeters + ) + if (lowered) { + this.lowerSince ??= sample.now + } else { this.lowerSince = null + } + + const turnedAwayLongEnough = this.turnAwaySince !== null + && sample.now - this.turnAwaySince + >= STATUS_GESTURE_THRESHOLDS.turnAwayHoldMs + const loweredLongEnough = this.lowerSince !== null + && sample.now - this.lowerSince + >= STATUS_GESTURE_THRESHOLDS.lowerHoldMs + if (turnedAwayLongEnough || loweredLongEnough) { + this.resetDismissalIntent() this.cooldownUntil = sample.now + STATUS_GESTURE_THRESHOLDS.cooldownMs return 'close' as const } return null } + this.wasOpen = false + this.openReferenceHeight = null + this.resetTrackingContinuity() const posed = sample.wristHeightFromEyes >= STATUS_GESTURE_THRESHOLDS.openHeightMeters && sample.handBackFacingViewer >= STATUS_GESTURE_THRESHOLDS.openFacingDot + && sample.gazeAtHandDot >= STATUS_GESTURE_THRESHOLDS.openGazeDot if (!posed || sample.now < this.cooldownUntil) { this.candidateSince = null return null @@ -296,6 +550,7 @@ export class StatusGestureModel { this.candidateSince ??= sample.now if (sample.now - this.candidateSince >= STATUS_GESTURE_THRESHOLDS.holdMs) { this.candidateSince = null + this.openReferenceHeight = sample.wristHeightFromEyes return 'open' as const } return null diff --git a/packages/webxr/src/status-window-placement.ts b/packages/webxr/src/status-window-placement.ts new file mode 100644 index 0000000..ffa9f36 --- /dev/null +++ b/packages/webxr/src/status-window-placement.ts @@ -0,0 +1,115 @@ +import { Vector3 } from 'three' + +export const STATUS_WINDOW_DISTANCE_METERS = 0.4 +export const STATUS_WINDOW_WRIST_RISE_METERS = 0.18 +export const STATUS_WINDOW_MAX_ANCHOR_JUMP_METERS = 0.12 +export const STATUS_WINDOW_REACQUIRE_STABILITY_METERS = 0.04 +export const STATUS_WINDOW_REACQUIRE_SECONDS = 0.12 +export const STATUS_WINDOW_FOLLOW_RATE = 12 + +const anchorDirection = new Vector3() + +export const resolveHandStatusWindowAnchorPosition = ( + wristPosition: Vector3, + target = new Vector3() +) => target.copy(wristPosition).setY( + wristPosition.y + STATUS_WINDOW_WRIST_RISE_METERS +) + +export const resolveStatusWindowAnchorPosition = ( + wristPosition: Vector3, + viewerPosition: Vector3, + target = new Vector3() +) => { + target.copy(wristPosition) + target.y += STATUS_WINDOW_WRIST_RISE_METERS + anchorDirection.subVectors(target, viewerPosition) + if (anchorDirection.lengthSq() < 0.0001) { + anchorDirection.set(-0.08, 0, -1) + } + return target.copy(viewerPosition).add( + anchorDirection.setLength(STATUS_WINDOW_DISTANCE_METERS) + ) +} + +/** + * Keeps a hand-opened status window at its last trustworthy world pose while + * the selecting hand is approaching or the reveal hand is occluded. Ordinary + * tracked motion remains smoothed and follows the left wrist. + */ +export class StatusWindowAnchorTracker { + private readonly position = new Vector3() + + private readonly acceptedTarget = new Vector3() + + private readonly candidate = new Vector3() + + private readonly reacquireCandidate = new Vector3() + + private hasPosition = false + + private hasAcceptedTarget = false + + private hasReacquireCandidate = false + + private reacquireSeconds = 0 + + reset() { + this.hasPosition = false + this.hasAcceptedTarget = false + this.hasReacquireCandidate = false + this.reacquireSeconds = 0 + } + + update(input: { + wristPosition: Vector3 | null + selectionEngaged: boolean + deltaSeconds: number + }) { + if (input.selectionEngaged || !input.wristPosition) { + this.hasReacquireCandidate = false + this.reacquireSeconds = 0 + return this.hasPosition ? this.position : null + } + + const candidate = resolveHandStatusWindowAnchorPosition( + input.wristPosition, + this.candidate + ) + if ( + this.hasAcceptedTarget + && candidate.distanceTo(this.acceptedTarget) + > STATUS_WINDOW_MAX_ANCHOR_JUMP_METERS + ) { + if ( + !this.hasReacquireCandidate + || candidate.distanceTo(this.reacquireCandidate) + > STATUS_WINDOW_REACQUIRE_STABILITY_METERS + ) { + this.reacquireCandidate.copy(candidate) + this.hasReacquireCandidate = true + this.reacquireSeconds = 0 + return this.hasPosition ? this.position : null + } + this.reacquireSeconds += Math.max(0, input.deltaSeconds) + if (this.reacquireSeconds < STATUS_WINDOW_REACQUIRE_SECONDS) { + return this.hasPosition ? this.position : null + } + } + this.hasReacquireCandidate = false + this.reacquireSeconds = 0 + this.acceptedTarget.copy(candidate) + this.hasAcceptedTarget = true + if (!this.hasPosition) { + this.position.copy(candidate) + this.hasPosition = true + return this.position + } + + const alpha = 1 - Math.exp( + -STATUS_WINDOW_FOLLOW_RATE * Math.max(0, input.deltaSeconds) + ) + this.position.lerp(candidate, alpha) + return this.position + } +} diff --git a/packages/webxr/src/status-window-view.ts b/packages/webxr/src/status-window-view.ts index d3ab1ac..cd56f73 100644 --- a/packages/webxr/src/status-window-view.ts +++ b/packages/webxr/src/status-window-view.ts @@ -10,6 +10,7 @@ import { } from 'three' import type { StatusAction, + StatusWindowPhase, StatusWindowSnapshot } from './status-window-model' import { @@ -19,15 +20,139 @@ import { } from './status-window-model' import { CanvasTextSurface } from './text-surface' -const WIDTH_METERS = 0.72 -const HEIGHT_METERS = 0.96 +export const STATUS_WINDOW_WIDTH_METERS = 0.24 +export const STATUS_WINDOW_HEIGHT_METERS = 0.32 +export const STATUS_WINDOW_CORNER_RADIUS_PIXELS = 48 +export const STATUS_WINDOW_APPEARANCE = { + background: 'rgba(43, 82, 8, 0.84)', + border: 'rgba(198, 255, 70, 0.96)', + accent: '#c8ff4d', + text: '#efffc2', + muted: 'rgba(222, 255, 166, 0.78)', + glow: 'rgba(194, 255, 67, 0.58)' +} as const +export const STATUS_WINDOW_ACTION_APPEARANCE = { + idleFill: 'rgba(177, 255, 61, 0.14)', + pressedFill: 'rgba(210, 255, 112, 0.42)', + pressedBorder: 'rgba(226, 255, 166, 0.96)' +} as const +export const STATUS_WINDOW_TOGGLE_APPEARANCE = { + onTrack: 'rgba(200, 255, 77, 0.88)', + offTrack: 'rgba(184, 220, 120, 0.2)', + disabledTrack: 'rgba(128, 153, 92, 0.12)', + knob: '#efffc2', + disabledKnob: 'rgba(220, 236, 188, 0.46)' +} as const + +const WIDTH_METERS = STATUS_WINDOW_WIDTH_METERS +const HEIGHT_METERS = STATUS_WINDOW_HEIGHT_METERS const WIDTH_PIXELS = 720 const HEIGHT_PIXELS = 960 const ACTION_TOP_PIXELS = 570 const ACTION_AREA_PIXELS = 310 +const HIT_INSET_METERS = 0.027 +export const STATUS_WINDOW_HIT_DEPTH_METERS = 0.004 +const HIT_DEPTH_METERS = STATUS_WINDOW_HIT_DEPTH_METERS const clamp01 = (value: number) => Math.min(1, Math.max(0, value)) +export const resolveStatusToggleLayout = ( + checked: boolean, + centerY: number +) => { + const track = { + x: WIDTH_PIXELS - 146, + y: centerY - 18, + width: 88, + height: 36, + radius: 18 + } + const knobSize = 28 + return { + track, + knob: { + x: checked + ? track.x + track.width - knobSize - 4 + : track.x + 4, + y: centerY - (knobSize / 2), + size: knobSize, + radius: knobSize / 2 + } + } +} + +export const drawStatusToggle = ( + context: Pick< + CanvasRenderingContext2D, + 'beginPath' | 'roundRect' | 'fill' | 'fillStyle' + >, + input: { + checked: boolean + available: boolean + centerY: number + } +) => { + const layout = resolveStatusToggleLayout(input.checked, input.centerY) + context.beginPath() + context.roundRect( + layout.track.x, + layout.track.y, + layout.track.width, + layout.track.height, + layout.track.radius + ) + context.fillStyle = input.available + ? input.checked + ? STATUS_WINDOW_TOGGLE_APPEARANCE.onTrack + : STATUS_WINDOW_TOGGLE_APPEARANCE.offTrack + : STATUS_WINDOW_TOGGLE_APPEARANCE.disabledTrack + context.fill() + context.beginPath() + context.roundRect( + layout.knob.x, + layout.knob.y, + layout.knob.size, + layout.knob.size, + layout.knob.radius + ) + context.fillStyle = input.available + ? STATUS_WINDOW_TOGGLE_APPEARANCE.knob + : STATUS_WINDOW_TOGGLE_APPEARANCE.disabledKnob + context.fill() +} + +export const drawStatusWindowFrame = ( + context: Pick< + CanvasRenderingContext2D, + | 'beginPath' + | 'roundRect' + | 'fill' + | 'stroke' + | 'fillStyle' + | 'strokeStyle' + | 'lineWidth' + | 'shadowColor' + | 'shadowBlur' + > +) => { + context.beginPath() + context.roundRect( + 12, + 12, + WIDTH_PIXELS - 24, + HEIGHT_PIXELS - 24, + STATUS_WINDOW_CORNER_RADIUS_PIXELS + ) + context.fillStyle = STATUS_WINDOW_APPEARANCE.background + context.fill() + context.strokeStyle = STATUS_WINDOW_APPEARANCE.border + context.lineWidth = 6 + context.shadowColor = STATUS_WINDOW_APPEARANCE.glow + context.shadowBlur = 28 + context.stroke() + context.shadowBlur = 0 +} + const formatReset = (value: string | null) => { if (!value) { return 'reset unavailable' @@ -47,7 +172,7 @@ const progress = ( remainingPercent: number | null, detail: string ) => { - context.fillStyle = '#dfffaa' + context.fillStyle = STATUS_WINDOW_APPEARANCE.accent context.font = '600 27px Inter, system-ui, sans-serif' context.fillText(label, 48, y) context.textAlign = 'right' @@ -59,10 +184,10 @@ const progress = ( y ) context.textAlign = 'left' - context.fillStyle = 'rgba(220, 255, 167, 0.18)' + context.fillStyle = 'rgba(197, 255, 77, 0.22)' context.fillRect(48, y + 18, WIDTH_PIXELS - 96, 14) if (remainingPercent != null) { - context.fillStyle = '#b9f46f' + context.fillStyle = STATUS_WINDOW_APPEARANCE.accent context.fillRect( 48, y + 18, @@ -70,7 +195,7 @@ const progress = ( 14 ) } - context.fillStyle = 'rgba(222, 255, 180, 0.66)' + context.fillStyle = STATUS_WINDOW_APPEARANCE.muted context.font = '22px Inter, system-ui, sans-serif' context.fillText(detail, 48, y + 62) } @@ -109,7 +234,9 @@ export class StatusWindowView { private snapshot: StatusWindowSnapshot | null = null - private phase: 'closed' | 'opening' | 'open' | 'closing' = 'closed' + private pressedAction: StatusAction['id'] | null = null + + private phase: StatusWindowPhase = 'closed' private phaseStartedAt = 0 @@ -166,14 +293,20 @@ export class StatusWindowView { while (this.actionHits.length < count) { const index = this.actionHits.length const hit = new Mesh( - new BoxGeometry(WIDTH_METERS - 0.08, rowPixels / HEIGHT_PIXELS * HEIGHT_METERS, 0.045), + new BoxGeometry( + WIDTH_METERS - HIT_INSET_METERS, + rowPixels / HEIGHT_PIXELS * HEIGHT_METERS, + HIT_DEPTH_METERS + ), new MeshBasicMaterial({ transparent: true, opacity: 0, depthWrite: false }) ) - hit.position.z = 0.025 + // Keep the touch volume behind the visible surface. A fingertip is a + // direct touch only when its tracked joint radius reaches z=0. + hit.position.z = -HIT_DEPTH_METERS / 2 hit.name = `status-action-${index}` this.actionHits.push(hit) this.windowGroup.add(hit) @@ -193,6 +326,10 @@ export class StatusWindowView { return this.phase === 'opening' || this.phase === 'open' } + get isFullyOpen() { + return this.phase === 'open' + } + setSnapshot(snapshot: StatusWindowSnapshot) { this.snapshot = snapshot this.reconcileActionHits(snapshot.actions.length) @@ -207,6 +344,17 @@ export class StatusWindowView { this.render() } + setPressedAction(action: StatusAction['id'] | null) { + if (action === null && this.phase === 'closing') { + return + } + if (this.pressedAction === action) { + return + } + this.pressedAction = action + this.render() + } + setMenuVisible(visible: boolean) { this.menuRequested = visible this.menuGroup.visible = visible && !this.isOpen @@ -251,6 +399,7 @@ export class StatusWindowView { this.windowGroup.scale.set(scale.x, scale.y, 1) if (progressValue >= 1) { this.phase = 'closed' + this.pressedAction = null this.group.visible = false this.menuGroup.visible = this.menuRequested } @@ -271,22 +420,43 @@ export class StatusWindowView { )[index]! const rowPixels = row.height const y = row.top + const pressed = action.available && action.id === this.pressedAction + context.fillStyle = pressed + ? STATUS_WINDOW_ACTION_APPEARANCE.pressedFill + : action.available + ? STATUS_WINDOW_ACTION_APPEARANCE.idleFill + : 'rgba(128, 153, 92, 0.08)' + context.beginPath() + context.roundRect(38, y + 4, WIDTH_PIXELS - 76, rowPixels - 8, 14) + context.fill() + if (pressed) { + context.strokeStyle = STATUS_WINDOW_ACTION_APPEARANCE.pressedBorder + context.lineWidth = 4 + context.shadowColor = STATUS_WINDOW_APPEARANCE.glow + context.shadowBlur = 18 + context.stroke() + context.shadowBlur = 0 + } context.fillStyle = action.available - ? 'rgba(177, 235, 96, 0.09)' - : 'rgba(125, 141, 100, 0.05)' - context.fillRect(38, y + 4, WIDTH_PIXELS - 76, rowPixels - 8) - context.fillStyle = action.available - ? '#e3ffb8' - : 'rgba(221, 236, 197, 0.48)' + ? STATUS_WINDOW_APPEARANCE.text + : 'rgba(220, 236, 188, 0.56)' context.font = '600 27px Inter, system-ui, sans-serif' context.fillText(action.label, 56, y + Math.min(39, rowPixels * 0.68)) - context.textAlign = 'right' - context.fillText( - action.available ? (action.state ?? '›') : 'Unavailable', - WIDTH_PIXELS - 56, - y + Math.min(39, rowPixels * 0.68) - ) - context.textAlign = 'left' + if (action.presentation === 'toggle' && action.checked !== null) { + drawStatusToggle(context, { + checked: action.checked, + available: action.available, + centerY: y + (rowPixels / 2) + }) + } else { + context.textAlign = 'right' + context.fillText( + action.available ? '›' : 'Unavailable', + WIDTH_PIXELS - 56, + y + Math.min(39, rowPixels * 0.68) + ) + context.textAlign = 'left' + } } private render() { @@ -296,18 +466,14 @@ export class StatusWindowView { return } context.clearRect(0, 0, WIDTH_PIXELS, HEIGHT_PIXELS) - context.fillStyle = 'rgba(44, 68, 16, 0.78)' - context.fillRect(0, 0, WIDTH_PIXELS, HEIGHT_PIXELS) - context.strokeStyle = 'rgba(194, 255, 116, 0.78)' - context.lineWidth = 5 - context.strokeRect(4, 4, WIDTH_PIXELS - 8, HEIGHT_PIXELS - 8) - context.shadowColor = 'rgba(187, 255, 105, 0.35)' - context.shadowBlur = 18 - context.fillStyle = '#ecffc9' + drawStatusWindowFrame(context) + context.shadowColor = STATUS_WINDOW_APPEARANCE.glow + context.shadowBlur = 16 + context.fillStyle = STATUS_WINDOW_APPEARANCE.text context.font = '700 42px Inter, system-ui, sans-serif' context.fillText('Codex status', 48, 68) context.shadowBlur = 0 - context.fillStyle = 'rgba(222, 255, 180, 0.7)' + context.fillStyle = STATUS_WINDOW_APPEARANCE.muted context.font = '23px Inter, system-ui, sans-serif' const identity = [snapshot.workspaceLabel, snapshot.threadLabel] .filter(Boolean).join(' · ') || 'Workspace identity unavailable' @@ -343,7 +509,7 @@ export class StatusWindowView { ? 'context usage unavailable' : `${Math.round(snapshot.context.remainingTokens).toLocaleString()} tokens remaining` ) - context.fillStyle = 'rgba(222, 255, 180, 0.72)' + context.fillStyle = STATUS_WINDOW_APPEARANCE.muted context.font = '23px Inter, system-ui, sans-serif' context.fillText( `${snapshot.connection} · ${snapshot.voice} · ${snapshot.activePaneCount} panes`, @@ -351,20 +517,20 @@ export class StatusWindowView { 420 ) context.fillText(snapshot.sessionLabel, 48, 454) - context.strokeStyle = 'rgba(194, 255, 116, 0.42)' + context.strokeStyle = 'rgba(198, 255, 70, 0.54)' context.lineWidth = 2 context.beginPath() context.moveTo(48, 478) context.lineTo(WIDTH_PIXELS - 48, 478) context.stroke() - context.fillStyle = '#dfffaa' + context.fillStyle = STATUS_WINDOW_APPEARANCE.accent context.font = '700 24px Inter, system-ui, sans-serif' context.fillText('ACTIONS', 48, 530) snapshot.actions.forEach((action, index) => { this.renderAction(context, action, index) }) const unavailable = snapshot.actions.find(action => !action.available) - context.fillStyle = 'rgba(222, 255, 180, 0.58)' + context.fillStyle = 'rgba(222, 255, 166, 0.68)' context.font = '20px Inter, system-ui, sans-serif' context.fillText( (unavailable?.disabledReason ?? 'Touch directly with a tracked index fingertip.').slice(0, 61), diff --git a/packages/webxr/src/transcript-bubble-view.ts b/packages/webxr/src/transcript-bubble-view.ts index 2fe15c1..b41dd3f 100644 --- a/packages/webxr/src/transcript-bubble-view.ts +++ b/packages/webxr/src/transcript-bubble-view.ts @@ -46,6 +46,12 @@ export const shouldRenderTranscriptTexture = (input: { ) ) +export const TRANSCRIPT_BUBBLE_APPEARANCE = { + background: 'rgba(12, 70, 96, 0.62)', + border: 'rgba(104, 225, 255, 0.96)', + text: '#d9f8ff' +} as const + export class TranscriptBubbleView { readonly group = new Group() @@ -54,9 +60,9 @@ export class TranscriptBubbleView { heightMeters: 0.62, widthPixels: 1_536, heightPixels: 560, - background: 'rgba(2, 16, 28, 0.2)', - border: 'rgba(65, 221, 255, 0.9)', - color: '#79e8ff', + background: TRANSCRIPT_BUBBLE_APPEARANCE.background, + border: TRANSCRIPT_BUBBLE_APPEARANCE.border, + color: TRANSCRIPT_BUBBLE_APPEARANCE.text, font: 'Inter, system-ui, "Noto Sans CJK KR", "Apple Color Emoji", sans-serif', lineHeightPixels: 52, paddingPixels: 60, diff --git a/packages/webxr/src/voice-runtime.ts b/packages/webxr/src/voice-runtime.ts index 0a747bc..be5dbbe 100644 --- a/packages/webxr/src/voice-runtime.ts +++ b/packages/webxr/src/voice-runtime.ts @@ -29,7 +29,7 @@ export type VoiceRuntimeOptions = { const DEFAULT_CONFIG_READ_TIMEOUT_MS = 5_000 -const voiceSessionActive = (snapshot: RealtimeConversationSnapshot) => +export const voiceSessionActive = (snapshot: RealtimeConversationSnapshot) => snapshot.state === 'requesting-permission' || snapshot.state === 'creating-offer' || snapshot.state === 'starting' diff --git a/packages/webxr/src/xr-capability.ts b/packages/webxr/src/xr-capability.ts index 798c8a4..dfea87d 100644 --- a/packages/webxr/src/xr-capability.ts +++ b/packages/webxr/src/xr-capability.ts @@ -23,7 +23,7 @@ export type ImmersiveModeSupport = { export type PassthroughAvailability = { supported: boolean active: boolean - contrast: 'dither' | 'additive-shape' | 'opaque' + contrast: 'dark-feather' | 'additive-shape' | 'opaque' disabledReason: string | null } @@ -71,7 +71,7 @@ export const resolvePassthroughAvailability = (input: { contrast: input.environmentBlendMode === 'additive' ? 'additive-shape' : input.environmentBlendMode === 'alpha-blend' - ? 'dither' + ? 'dark-feather' : 'opaque', disabledReason: 'Immersive VR is unavailable; exit immersive to leave AR.' } @@ -89,7 +89,7 @@ export const resolvePassthroughAvailability = (input: { active: true, contrast: input.environmentBlendMode === 'additive' ? 'additive-shape' - : 'dither', + : 'dark-feather', disabledReason: null } } diff --git a/packages/webxr/test/hand-outline-view.test.ts b/packages/webxr/test/hand-outline-view.test.ts new file mode 100644 index 0000000..93961a8 --- /dev/null +++ b/packages/webxr/test/hand-outline-view.test.ts @@ -0,0 +1,91 @@ +import { describe, expect, it, vi } from 'vitest' +import { + Matrix4, + MeshBasicMaterial, + Quaternion, + Raycaster, + Vector3 +} from 'three' +import { + createDevelopmentHandPose, + HAND_BONE_CONNECTIONS, + HAND_JOINT_NAMES, + HandOutlineView, + resolveHandJointRadius +} from '../src/hand-outline-view' + +describe('tracked hand outline', () => { + it('uses one physical joint radius for rendering and touch collision', () => { + expect(resolveHandJointRadius(0.009)).toBe(0.009) + expect(resolveHandJointRadius()).toBe(0.006) + }) + + it('renders a connected, handed joint silhouette without hit testing', () => { + const view = new HandOutlineView('left') + view.update(createDevelopmentHandPose('left')) + + expect(view.group.name).toBe('tracked-hand-outline-left') + expect(view.group.visible).toBe(true) + expect(view.innerBones.count).toBe(HAND_BONE_CONNECTIONS.length) + expect(view.innerJoints.count).toBe(HAND_JOINT_NAMES.length) + expect(view.outerBones.count).toBe(view.innerBones.count) + expect(view.outerJoints.count).toBe(view.innerJoints.count) + + const inner = view.innerBones.material as MeshBasicMaterial + const outer = view.outerBones.material as MeshBasicMaterial + expect(inner).toMatchObject({ + transparent: true, + opacity: 0.52, + depthWrite: false, + depthTest: true, + toneMapped: false + }) + expect(outer.opacity).toBeLessThan(inner.opacity) + expect(inner.color.getHexString()).toBe('a8efff') + expect(view.group.userData.representationalOnly).toBe(true) + + const fingertipMatrix = new Matrix4() + const fingertipScale = new Vector3() + view.outerJoints.getMatrixAt( + HAND_JOINT_NAMES.indexOf('index-finger-tip'), + fingertipMatrix + ) + fingertipMatrix.decompose( + new Vector3(), + new Quaternion(), + fingertipScale + ) + expect(fingertipScale.x).toBeCloseTo( + createDevelopmentHandPose('left').get('index-finger-tip')!.radius + ) + + view.group.updateMatrixWorld(true) + const raycaster = new Raycaster( + new Vector3(0, 0, 1), + new Vector3(0, 0, -1) + ) + expect(raycaster.intersectObject(view.group, true)).toHaveLength(0) + view.dispose() + }) + + it('updates handedness and hides cleanly when tracking is unavailable', () => { + const view = new HandOutlineView() + view.setHandedness('right') + view.update(createDevelopmentHandPose('right')) + expect(view.handedness).toBe('right') + expect(view.group.name).toBe('tracked-hand-outline-right') + expect( + (view.innerBones.material as MeshBasicMaterial).color.getHexString() + ).toBe('d8ffad') + + const disposeBoneGeometry = vi.spyOn(view.innerBones.geometry, 'dispose') + const disposeJointGeometry = vi.spyOn(view.innerJoints.geometry, 'dispose') + view.clear() + expect(view.group.visible).toBe(false) + expect(view.innerBones.count).toBe(0) + expect(view.innerJoints.count).toBe(0) + view.dispose() + expect(disposeBoneGeometry).toHaveBeenCalledTimes(1) + expect(disposeJointGeometry).toHaveBeenCalledTimes(1) + }) +}) diff --git a/packages/webxr/test/immersive-scene.test.ts b/packages/webxr/test/immersive-scene.test.ts new file mode 100644 index 0000000..b2e12fd --- /dev/null +++ b/packages/webxr/test/immersive-scene.test.ts @@ -0,0 +1,69 @@ +import { describe, expect, it } from 'vitest' +import { + GridHelper, + Mesh, + MeshBasicMaterial, + PlaneGeometry +} from 'three' +import { + configureRoomSurfaceRendering, + resolveWorldControlTargets, + shouldShowHandOutlines, + ROOM_FLOOR_RENDER_ORDER, + ROOM_GRID_RENDER_ORDER, + ROOM_WALL_RENDER_ORDER +} from '../src/immersive-scene' + +describe('immersive room rendering', () => { + it('hides synthetic hand outlines only in real passthrough modes', () => { + expect(shouldShowHandOutlines('immersive-vr', 'opaque')).toBe(true) + expect(shouldShowHandOutlines('immersive-ar', 'opaque')).toBe(true) + expect(shouldShowHandOutlines('immersive-ar', 'alpha-blend')).toBe(false) + expect(shouldShowHandOutlines('immersive-ar', 'additive')).toBe(false) + }) + + it('removes the voice orb from control targets while voice is active', () => { + const voice = new Mesh() + const exit = new Mesh() + + expect(resolveWorldControlTargets(true, voice, [exit])) + .toEqual([voice, exit]) + expect(resolveWorldControlTargets(false, voice, [exit])) + .toEqual([exit]) + }) + + it('renders floor surfaces before panes without writing occluding depth', () => { + const floorMaterial = new MeshBasicMaterial() + const floor = new Mesh(new PlaneGeometry(20, 20), floorMaterial) + const grid = new GridHelper(20, 20) + const wallMaterial = new MeshBasicMaterial() + const wall = new Mesh(new PlaneGeometry(20, 3.6), wallMaterial) + + configureRoomSurfaceRendering(floor, grid, [wall]) + + expect(floorMaterial.depthWrite).toBe(false) + for (const material of Array.isArray(grid.material) + ? grid.material + : [grid.material]) { + expect(material.depthWrite).toBe(false) + } + expect(floor.renderOrder).toBe(ROOM_FLOOR_RENDER_ORDER) + expect(grid.renderOrder).toBe(ROOM_GRID_RENDER_ORDER) + expect(wallMaterial.depthWrite).toBe(false) + expect(wall.renderOrder).toBe(ROOM_WALL_RENDER_ORDER) + expect(floor.renderOrder).toBeLessThan(0) + expect(grid.renderOrder).toBeLessThan(0) + expect(wall.renderOrder).toBeLessThan(0) + + floor.geometry.dispose() + floorMaterial.dispose() + wall.geometry.dispose() + wallMaterial.dispose() + grid.geometry.dispose() + for (const material of Array.isArray(grid.material) + ? grid.material + : [grid.material]) { + material.dispose() + } + }) +}) diff --git a/packages/webxr/test/pane-manipulation.test.ts b/packages/webxr/test/pane-manipulation.test.ts index eeb7b3e..9e767f1 100644 --- a/packages/webxr/test/pane-manipulation.test.ts +++ b/packages/webxr/test/pane-manipulation.test.ts @@ -8,6 +8,7 @@ import { PreferredPaneInputModel, resolveAcceleratedPaneDepth, resolveDepthLockedPanePosition, + resolveHandPinchPanePosition, resolveHandScrollRate, resolvePaneDepthActivationOffset, resolvePrimaryThumbstickY, @@ -108,6 +109,106 @@ describe('pane manipulation helpers', () => { expect(later.z).toBeLessThan(freeAtActivation.z - 0.2) }) + it('keeps live hand-pinch lateral coordinates while accelerating depth', () => { + const initialViewerPosition = new Vector3(0, 1.6, 0) + const initialPanelPosition = new Vector3(0, 1.6, -2) + const initialSourcePosition = new Vector3(0, 1.6, -0.5) + const sightLine = new Vector3(0, 0, -1) + const activation = new Vector3(0.01, -0.005, 0.06) + const activationPhysicalDepth = activation.dot(sightLine) + const atActivation = resolveHandPinchPanePosition({ + initialPanelPosition, + initialViewerPosition, + initialSourcePosition, + sightLine, + sourceDisplacement: activation, + activationPhysicalDepth, + activationSourceDisplacement: activation, + accelerateDepth: true + }) + const expectedAtActivation = initialPanelPosition.clone().add(activation) + expect(atActivation.x).toBeCloseTo(expectedAtActivation.x) + expect(atActivation.y).toBeCloseTo(expectedAtActivation.y) + expect(atActivation.z).toBeCloseTo(expectedAtActivation.z) + + const moved = new Vector3(0.24, 0.08, 0.16) + const later = resolveHandPinchPanePosition({ + initialPanelPosition, + initialViewerPosition, + initialSourcePosition, + sightLine, + sourceDisplacement: moved, + activationPhysicalDepth, + activationSourceDisplacement: activation, + accelerateDepth: true + }) + const resolvedDepth = resolveAcceleratedPaneDepth({ + initialDistance: 2, + physicalDepth: moved.dot(sightLine), + activationPhysicalDepth + }) + const currentSourceDistance = initialSourcePosition.clone() + .add(moved) + .distanceTo(initialViewerPosition) + const lateralScale = resolvedDepth / currentSourceDistance + expect(later.x).toBeCloseTo( + activation.x + ((moved.x - activation.x) * lateralScale) + ) + expect(later.y).toBeCloseTo( + initialPanelPosition.y + activation.y + + ((moved.y - activation.y) * lateralScale) + ) + expect(later.z).toBeGreaterThan(initialPanelPosition.z + moved.z) + }) + + it('increases hand-pinch lateral gain as a pane moves farther away', () => { + const initialViewerPosition = new Vector3() + const initialPanelPosition = new Vector3(0, 0, -1) + const initialSourcePosition = new Vector3(0, 0, -0.5) + const sightLine = new Vector3(0, 0, -1) + const activation = new Vector3(0, 0, -0.05) + const activationPhysicalDepth = activation.dot(sightLine) + const resolve = (sourceDisplacement: Vector3) => ( + resolveHandPinchPanePosition({ + initialPanelPosition, + initialViewerPosition, + initialSourcePosition, + sightLine, + sourceDisplacement, + activationPhysicalDepth, + activationSourceDisplacement: activation, + accelerateDepth: true + }) + ) + + const near = resolve(new Vector3(0.1, 0, -0.05)) + const far = resolve(new Vector3(0.1, 0, -0.45)) + expect(near.x).toBeGreaterThan(0.1) + expect(far.x).toBeGreaterThan(near.x) + }) + + it('scales lateral hand-pinch movement without accelerating free depth', () => { + const initialViewerPosition = new Vector3() + const initialPanelPosition = new Vector3(0, 0, -2) + const initialSourcePosition = new Vector3(0, 0, -0.5) + const sightLine = new Vector3(0, 0, -1) + const activation = new Vector3(0.05, 0, 0) + const moved = new Vector3(0.15, 0, -0.1) + const result = resolveHandPinchPanePosition({ + initialPanelPosition, + initialViewerPosition, + initialSourcePosition, + sightLine, + sourceDisplacement: moved, + activationPhysicalDepth: 0, + activationSourceDisplacement: activation, + accelerateDepth: false + }) + + expect(result.x).toBeGreaterThan(moved.x) + expect(result.z).toBeCloseTo(initialPanelPosition.z + moved.z) + }) + it('maps only the right xr-standard primary thumbstick Y axis', () => { expect(resolvePrimaryThumbstickY(inputSource({ axes: [0.8, -0.8, 0.2, 0.65, -0.95] diff --git a/packages/webxr/test/panel-interaction.test.ts b/packages/webxr/test/panel-interaction.test.ts index ae9945b..377cb40 100644 --- a/packages/webxr/test/panel-interaction.test.ts +++ b/packages/webxr/test/panel-interaction.test.ts @@ -7,7 +7,6 @@ import { MeshBasicMaterial, PerspectiveCamera, Ray, - Sphere, Vector3, type WebGLRenderer, type XRHandSpace, @@ -20,12 +19,13 @@ import { resolveFocusedPanelLocalPosition, resolveFocusedPanelPosition, resolveRayPanelPosition, - resolveRayGrabPosition, resolveStatusFallbackMenuVisibility, + statusTargetIntersectsSphere, worldPointToPanelLocal, resolveTrackedHandJoint } from '../src/interaction-system' import { PanelInteractionModel } from '../src/panel-interaction' +import { createDevelopmentHandPose } from '../src/hand-outline-view' import type { SpatialPanelView } from '../src/panel-view' const createPanelDouble = (id = 'panel-1') => { @@ -57,6 +57,7 @@ const createPanelDouble = (id = 'panel-1') => { maximumScrollStart: 8, setInteraction: vi.fn(), setHandControlsVisible: vi.fn(), + setRayControlsVisible: vi.fn(), moveTo: (position: Vector3) => { group.position.copy(position) group.updateMatrixWorld(true) @@ -66,7 +67,15 @@ const createPanelDouble = (id = 'panel-1') => { } const createInteractionHarness = ( - panels: ReadonlyMap = new Map() + panels: ReadonlyMap = new Map(), + statusOptions: { + targets?: Mesh[] + state?: { + open: boolean + fullyOpen: boolean + invocation: 'controller' | 'hand' | 'fallback' | null + } + } = {} ) => { const targetRays = [new Group(), new Group()] const grips = [new Group(), new Group()] @@ -89,30 +98,53 @@ const createInteractionHarness = ( for (const panel of panels.values()) { root.add(panel.group) } + for (const target of statusOptions.targets ?? []) { + root.add(target) + } + const statusState = statusOptions.state ?? { + open: false, + fullyOpen: false, + invocation: null + } const callbacks = { onScroll: vi.fn(), onPanelInteracted: vi.fn(), onPanelMoved: vi.fn(), onPanelFocused: vi.fn(), - onPanelDismiss: vi.fn() + onPanelDismiss: vi.fn(), + onStatusToggle: vi.fn(), + onStatusDismiss: vi.fn(), + onStatusAction: vi.fn(), + onStatusPressedChanged: vi.fn() } const system = new ImmersiveInteractionSystem({ renderer, root, getPanels: () => panels, getControlTargets: () => [], - getStatusTargets: () => [], + getStatusTargets: () => statusOptions.targets ?? [], getStatusMenuTarget: () => null, - isStatusOpen: () => false, - getStatusInvocation: () => null, + isStatusOpen: () => statusState.open, + isStatusFullyOpen: () => statusState.fullyOpen, + getStatusInvocation: () => statusState.invocation, ...callbacks, onAction: () => {}, - onStatusToggle: () => {}, - onStatusDismiss: () => {}, - onStatusAction: () => {}, onInputCapabilitiesChanged: () => {} }) - return { system, targetRays, grips, hands, callbacks } + return { system, targetRays, grips, hands, callbacks, statusState } +} + +const createStatusActionTarget = (id = 'passthrough') => { + const target = new Mesh( + new BoxGeometry(0.42, 0.18, 0.05), + new MeshBasicMaterial({ transparent: true, opacity: 0 }) + ) + target.position.set(0, 0, -1) + target.userData.statusActionId = id + target.userData.statusActionAvailable = true + target.userData.statusInputPolicy = 'controller-or-touch' + target.updateMatrixWorld(true) + return target } const attachTrackedIndexTip = ( @@ -136,30 +168,52 @@ const attachTrackedIndexTip = ( return index } -describe('panel interaction model', () => { - it('keeps ray-grabbed pointer positions at a fixed viewer distance', () => { - const sphere = new Sphere(new Vector3(0, 0, 0), 2) - const offset = new Vector3() - const first = resolveRayGrabPosition( - new Ray( - new Vector3(0, 0, 0), - new Vector3(0.25, 0, -1).normalize() - ), - sphere, - offset - )! - const second = resolveRayGrabPosition( - new Ray( - new Vector3(0, 0, 0), - new Vector3(0.5, 0, -1).normalize() - ), - sphere, - offset - )! +describe('input capability controls', () => { + it('enables ray scroll controls for a controller-only setup', () => { + const { panel } = createPanelDouble() + const { system } = createInteractionHarness( + new Map([['panel-1', panel]]) + ) + const internals = system as unknown as { + sources: Array<{ inputSource: XRInputSource | null }> + updateInputCapabilities: () => void + } + internals.sources[0]!.inputSource = { + handedness: 'right', + hand: null, + targetRayMode: 'tracked-pointer' + } as unknown as XRInputSource + + internals.updateInputCapabilities() - expect(first.length()).toBeCloseTo(2) - expect(second.length()).toBeCloseTo(2) - expect(second.x).toBeGreaterThan(first.x) + expect(panel.setRayControlsVisible).toHaveBeenCalledWith(true) + expect(panel.setHandControlsVisible).toHaveBeenCalledWith(false) + system.dispose() + }) +}) + +describe('panel interaction model', () => { + it('uses the oriented panel box for fingertip sphere collision', () => { + const target = new Mesh( + new BoxGeometry(0.2, 0.1, 0.004), + new MeshBasicMaterial({ transparent: true, opacity: 0 }) + ) + target.position.set(0, 0, -1) + target.rotation.y = Math.PI / 4 + target.updateMatrixWorld(true) + const beforeSurface = target.localToWorld(new Vector3(0, 0, 0.03)) + const touchingSurface = target.localToWorld(new Vector3(0, 0, 0.008)) + + expect(statusTargetIntersectsSphere( + target, + beforeSurface, + 0.009 + )).toBe(false) + expect(statusTargetIntersectsSphere( + target, + touchingSurface, + 0.009 + )).toBe(true) }) it('treats only sub-classification movement as a focus tap', () => { @@ -330,6 +384,68 @@ describe('panel interaction model', () => { expect(resolveTrackedHandJoint(hand, 'wrist')).toBe(null) }) + it('shows tracked hand outlines and hides them for same-hand controllers or loss', () => { + const { system, hands } = createInteractionHarness() + const internals = system as unknown as { + sources: Array<{ + listeners: { + connected: (event: unknown) => void + disconnected: () => void + } + }> + } + const handRuntime = internals.sources[0]! + const controllerRuntime = internals.sources[1]! + handRuntime.listeners.connected({ + data: { + handedness: 'left', + hand: {}, + targetRayMode: 'tracked-pointer', + profiles: ['generic-hand-select'] + } as unknown as XRInputSource + }) + hands[0]!.visible = true + for (const [name, pose] of createDevelopmentHandPose('left')) { + const joint = Object.assign(new Group(), { + jointRadius: pose.radius + }) as unknown as XRJointSpace + joint.position.copy(pose.position) + joint.visible = true + hands[0]!.joints[name] = joint + hands[0]!.add(joint) + } + hands[0]!.updateMatrixWorld(true) + system.update(0, 1 / 60) + const outline = hands[0]!.getObjectByName('tracked-hand-outline-left')! + expect(outline.visible).toBe(true) + + system.setHandOutlinesEnabled(false) + expect(outline.visible).toBe(false) + system.update(0.5, 1 / 60) + expect(outline.visible).toBe(false) + system.setHandOutlinesEnabled(true) + system.update(0.75, 1 / 60) + expect(outline.visible).toBe(true) + + controllerRuntime.listeners.connected({ + data: { + handedness: 'left', + hand: null, + targetRayMode: 'tracked-pointer', + profiles: ['oculus-touch-v3'] + } as unknown as XRInputSource + }) + system.update(1, 1 / 60) + expect(outline.visible).toBe(false) + + controllerRuntime.listeners.disconnected() + system.update(2, 1 / 60) + expect(outline.visible).toBe(true) + handRuntime.listeners.disconnected() + expect(outline.visible).toBe(false) + system.dispose() + }) + it('recognizes a menu controller only when its mapped button is exposed', () => { const source = { handedness: 'left', @@ -415,6 +531,403 @@ describe('panel interaction model', () => { ])).toBe(false) }) + it('requires a stable left hand-back dwell and keeps the wrist anchor live', () => { + const { system, hands, callbacks } = createInteractionHarness() + const internals = system as unknown as { + sources: Array<{ + inputSource: XRInputSource | null + listeners: { connected: (event: unknown) => void } + }> + statusAnchor: () => Group | null + } + const runtime = internals.sources[0]! + runtime.listeners.connected({ + data: { + handedness: 'left', + hand: {}, + targetRayMode: 'tracked-pointer', + profiles: ['generic-hand-select'] + } as unknown as XRInputSource + }) + const wrist = Object.assign(new Group(), { + jointRadius: 0.014 + }) as unknown as XRJointSpace + wrist.position.set(0, -0.15, -0.5) + wrist.quaternion.setFromUnitVectors( + new Vector3(0, 1, 0), + wrist.position.clone().negate().normalize() + ) + wrist.visible = true + hands[0]!.visible = true + hands[0]!.joints.wrist = wrist + hands[0]!.add(wrist) + hands[0]!.updateMatrixWorld(true) + + system.update(0, 1 / 60) + system.update(699, 1 / 60) + expect(callbacks.onStatusToggle).not.toHaveBeenCalled() + system.update(700, 1 / 60) + expect(callbacks.onStatusToggle).toHaveBeenCalledTimes(1) + expect(callbacks.onStatusToggle).toHaveBeenCalledWith('hand') + + const anchoredBefore = internals.statusAnchor()! + .getWorldPosition(new Vector3()) + wrist.position.x = 0.18 + hands[0]!.updateMatrixWorld(true) + const anchoredAfter = internals.statusAnchor()! + .getWorldPosition(new Vector3()) + expect(anchoredAfter.x - anchoredBefore.x).toBeCloseTo(0.18) + system.dispose() + }) + + it('keeps a pre-existing right index overlap inert until a fresh contact', () => { + const target = createStatusActionTarget() + const statusState = { + open: true, + fullyOpen: false, + invocation: 'hand' as const + } + const { system, hands, callbacks } = createInteractionHarness( + new Map(), + { targets: [target], state: statusState } + ) + const internals = system as unknown as { + sources: Array<{ + listeners: { connected: (event: unknown) => void } + }> + } + internals.sources[0]!.listeners.connected({ + data: { + handedness: 'right', + hand: {}, + targetRayMode: 'tracked-pointer', + profiles: ['generic-hand-select'] + } as unknown as XRInputSource + }) + const index = Object.assign(new Group(), { + jointRadius: 0.01 + }) as unknown as XRJointSpace + index.position.copy(target.position) + index.visible = true + hands[0]!.visible = true + hands[0]!.joints['index-finger-tip'] = index + hands[0]!.add(index) + hands[0]!.updateMatrixWorld(true) + + system.update(0, 1 / 60) + statusState.fullyOpen = true + system.update(160, 1 / 60) + system.update(340, 1 / 60) + system.update(341, 1 / 60) + expect(callbacks.onStatusAction).not.toHaveBeenCalled() + + index.position.x = 1 + hands[0]!.updateMatrixWorld(true) + system.update(342, 1 / 60) + index.position.x = 0 + hands[0]!.updateMatrixWorld(true) + system.update(343, 1 / 60) + system.update(344, 1 / 60) + expect(callbacks.onStatusAction).toHaveBeenCalledTimes(1) + expect(callbacks.onStatusAction).toHaveBeenCalledWith('passthrough') + system.dispose() + }) + + it('treats approach as engagement but activates only at fingertip surface contact', () => { + const target = new Mesh( + new BoxGeometry(0.2, 0.1, 0.004), + new MeshBasicMaterial({ transparent: true, opacity: 0 }) + ) + target.position.set(0, 0, -1.002) + target.userData.statusActionId = 'passthrough' + target.userData.statusActionAvailable = true + target.userData.statusInputPolicy = 'controller-or-touch' + target.updateMatrixWorld(true) + const statusState = { + open: true, + fullyOpen: true, + invocation: 'hand' as const + } + const { system, hands, callbacks } = createInteractionHarness( + new Map(), + { targets: [target], state: statusState } + ) + const internals = system as unknown as { + sources: Array<{ + listeners: { connected: (event: unknown) => void } + }> + } + internals.sources[0]!.listeners.connected({ + data: { + handedness: 'right', + hand: {}, + targetRayMode: 'tracked-pointer', + profiles: ['generic-hand-select'] + } as unknown as XRInputSource + }) + const index = Object.assign(new Group(), { + jointRadius: 0.009 + }) as unknown as XRJointSpace + index.position.set(0, 0, -0.97) + index.visible = true + hands[0]!.visible = true + hands[0]!.joints['index-finger-tip'] = index + hands[0]!.add(index) + hands[0]!.updateMatrixWorld(true) + + system.update(0, 1 / 60) + system.update(180, 1 / 60) + system.update(181, 1 / 60) + expect(system.isStatusHandEngaged()).toBe(true) + expect(callbacks.onStatusAction).not.toHaveBeenCalled() + expect(callbacks.onStatusPressedChanged).not.toHaveBeenCalledWith( + 'passthrough' + ) + + index.position.z = -0.992 + hands[0]!.updateMatrixWorld(true) + system.update(182, 1 / 60) + expect(callbacks.onStatusAction).toHaveBeenCalledTimes(1) + expect(callbacks.onStatusAction).toHaveBeenCalledWith('passthrough') + expect(callbacks.onStatusPressedChanged).toHaveBeenCalledWith( + 'passthrough' + ) + system.dispose() + }) + + it('ignores left-hand contact and pinch for status actions', () => { + const target = createStatusActionTarget() + const statusState = { + open: true, + fullyOpen: true, + invocation: 'hand' as const + } + const { system, hands, callbacks } = createInteractionHarness( + new Map(), + { targets: [target], state: statusState } + ) + const internals = system as unknown as { + sources: Array<{ + listeners: { + connected: (event: unknown) => void + selectstart: () => void + selectend: () => void + } + }> + } + internals.sources[0]!.listeners.connected({ + data: { + handedness: 'left', + hand: {}, + targetRayMode: 'tracked-pointer', + profiles: ['generic-hand-select'] + } as unknown as XRInputSource + }) + const index = Object.assign(new Group(), { + jointRadius: 0.01 + }) as unknown as XRJointSpace + index.position.set(1, 0, -1) + index.visible = true + hands[0]!.visible = true + hands[0]!.joints['index-finger-tip'] = index + hands[0]!.add(index) + system.update(0, 1 / 60) + system.update(180, 1 / 60) + index.position.x = 0 + hands[0]!.updateMatrixWorld(true) + system.update(181, 1 / 60) + internals.sources[0]!.listeners.selectstart() + internals.sources[0]!.listeners.selectend() + expect(callbacks.onStatusAction).not.toHaveBeenCalled() + system.dispose() + }) + + it('does not reinterpret a right-hand pinch as status direct touch', () => { + const target = createStatusActionTarget() + const statusState = { + open: true, + fullyOpen: true, + invocation: 'hand' as const + } + const { system, hands, callbacks } = createInteractionHarness( + new Map(), + { targets: [target], state: statusState } + ) + const internals = system as unknown as { + sources: Array<{ + listeners: { connected: (event: unknown) => void } + }> + } + internals.sources[0]!.listeners.connected({ + data: { + handedness: 'right', + hand: {}, + targetRayMode: 'tracked-pointer', + profiles: ['generic-hand-select'] + } as unknown as XRInputSource + }) + const index = Object.assign(new Group(), { + jointRadius: 0.01 + }) as unknown as XRJointSpace + const thumb = Object.assign(new Group(), { + jointRadius: 0.01 + }) as unknown as XRJointSpace + index.position.set(1, 0, -1) + thumb.position.set(1.08, 0, -1) + index.visible = true + thumb.visible = true + hands[0]!.visible = true + hands[0]!.joints['index-finger-tip'] = index + hands[0]!.joints['thumb-tip'] = thumb + hands[0]!.add(index, thumb) + hands[0]!.updateMatrixWorld(true) + system.update(0, 1 / 60) + system.update(180, 1 / 60) + + index.position.set(0, 0, -1) + thumb.position.set(0.02, 0, -1) + hands[0]!.updateMatrixWorld(true) + system.update(181, 1 / 60) + expect(callbacks.onStatusAction).not.toHaveBeenCalled() + + thumb.position.set(0.08, 0, -1) + hands[0]!.updateMatrixWorld(true) + system.update(182, 1 / 60) + expect(callbacks.onStatusAction).not.toHaveBeenCalled() + + index.position.x = 1 + hands[0]!.updateMatrixWorld(true) + system.update(183, 1 / 60) + index.position.x = 0 + hands[0]!.updateMatrixWorld(true) + system.update(184, 1 / 60) + expect(callbacks.onStatusAction).toHaveBeenCalledTimes(1) + system.dispose() + }) + + it('requires release after a held right-controller press, then activates once', () => { + const target = createStatusActionTarget() + const statusState = { + open: true, + fullyOpen: false, + invocation: 'controller' as const + } + const { system, callbacks } = createInteractionHarness( + new Map(), + { targets: [target], state: statusState } + ) + const internals = system as unknown as { + sources: Array<{ + listeners: { + connected: (event: unknown) => void + selectstart: () => void + selectend: () => void + } + }> + } + const runtime = internals.sources[0]! + runtime.listeners.connected({ + data: { + handedness: 'right', + hand: null, + targetRayMode: 'tracked-pointer', + profiles: ['oculus-touch-v3'] + } as unknown as XRInputSource + }) + system.update(0, 1 / 60) + runtime.listeners.selectstart() + expect(callbacks.onStatusAction).not.toHaveBeenCalled() + + statusState.fullyOpen = true + system.update(160, 1 / 60) + system.update(340, 1 / 60) + runtime.listeners.selectstart() + expect(callbacks.onStatusAction).not.toHaveBeenCalled() + + runtime.listeners.selectend() + runtime.listeners.selectstart() + runtime.listeners.selectstart() + expect(callbacks.onStatusAction).toHaveBeenCalledTimes(1) + expect(callbacks.onStatusAction).toHaveBeenCalledWith('passthrough') + system.dispose() + }) + + it('keeps controller menu toggling edge-triggered', () => { + const { system, callbacks } = createInteractionHarness() + const button = { pressed: false } + const internals = system as unknown as { + sources: Array<{ + listeners: { connected: (event: unknown) => void } + }> + } + internals.sources[0]!.listeners.connected({ + data: { + handedness: 'left', + hand: null, + targetRayMode: 'tracked-pointer', + profiles: ['htc-vive-focus'], + gamepad: { + buttons: [{}, {}, {}, {}, button] + } + } as unknown as XRInputSource + }) + system.update(0, 1 / 60) + button.pressed = true + system.update(1, 1 / 60) + system.update(2, 1 / 60) + expect(callbacks.onStatusToggle).toHaveBeenCalledTimes(1) + expect(callbacks.onStatusToggle).toHaveBeenCalledWith('controller') + system.dispose() + }) + + it('dismisses a deliberately lowered left hand without firing an action', () => { + const statusState = { + open: true, + fullyOpen: true, + invocation: 'hand' as const + } + const { system, hands, callbacks } = createInteractionHarness( + new Map(), + { state: statusState } + ) + const internals = system as unknown as { + sources: Array<{ + listeners: { connected: (event: unknown) => void } + }> + } + internals.sources[0]!.listeners.connected({ + data: { + handedness: 'left', + hand: {}, + targetRayMode: 'tracked-pointer', + profiles: ['generic-hand-select'] + } as unknown as XRInputSource + }) + const wrist = Object.assign(new Group(), { + jointRadius: 0.014 + }) as unknown as XRJointSpace + wrist.position.set(0, -0.15, -0.5) + wrist.visible = true + hands[0]!.visible = true + hands[0]!.joints.wrist = wrist + hands[0]!.add(wrist) + hands[0]!.updateMatrixWorld(true) + system.update(0, 1 / 60) + system.update(180, 1 / 60) + wrist.position.y = -0.3 + hands[0]!.updateMatrixWorld(true) + system.update(280, 1 / 60) + wrist.position.y = -0.4 + hands[0]!.updateMatrixWorld(true) + system.update(380, 1 / 60) + system.update(699, 1 / 60) + expect(callbacks.onStatusDismiss).not.toHaveBeenCalled() + system.update(700, 1 / 60) + expect(callbacks.onStatusDismiss).toHaveBeenCalledTimes(1) + expect(callbacks.onStatusAction).not.toHaveBeenCalled() + system.dispose() + }) + it.each(['thumb-tip', 'index-finger-tip'] as const)( 'ends an active synthesized pinch once when %s tracking is lost', (lostJointName) => { @@ -539,6 +1052,109 @@ describe('panel interaction model', () => { system.dispose() }) + it.each([ + ['remote pinch cursor', false], + ['controller ray', true] + ] as const)('scrolls up once when the %s selects the arrow', (_label, native) => { + const { panel, dismissHit, scrollUpHit } = createPanelDouble() + dismissHit.visible = false + scrollUpHit.visible = true + scrollUpHit.position.z = 0.045 + const { system, callbacks } = createInteractionHarness( + new Map([['panel-1', panel]]) + ) + const internals = system as unknown as { + sources: unknown[] + handleSelectStart: ( + runtime: unknown, + now: number, + native: boolean + ) => void + } + + internals.handleSelectStart(internals.sources[0], 0, native) + expect(callbacks.onPanelInteracted).toHaveBeenCalledWith('panel-1') + expect(callbacks.onScroll).toHaveBeenCalledWith('panel-1', -1, 8) + expect(callbacks.onPanelMoved).not.toHaveBeenCalled() + system.dispose() + }) + + it('pulls a pane toward the viewer from physical hand-pinch motion', () => { + const { panel, dismissHit, group } = createPanelDouble() + dismissHit.visible = false + const { system, hands, callbacks } = createInteractionHarness( + new Map([['panel-1', panel]]) + ) + const internals = system as unknown as { + sources: Array<{ + pinching: boolean + grabbedBy: 'pinch' | null + grabIntent: 'neutral' | 'free' | 'depth' + listeners: { connected: (event: unknown) => void } + }> + } + const runtime = internals.sources[0]! + runtime.listeners.connected({ + data: { + handedness: 'right', + hand: {}, + targetRayMode: 'tracked-pointer', + profiles: ['generic-hand-select'] + } as unknown as XRInputSource + }) + const thumb = Object.assign(new Group(), { + jointRadius: 0.009 + }) as unknown as XRJointSpace + const index = Object.assign(new Group(), { + jointRadius: 0.009 + }) as unknown as XRJointSpace + thumb.position.set(0.01, 0, -0.5) + index.position.set(-0.01, 0, -0.5) + thumb.visible = true + index.visible = true + hands[0]!.visible = true + hands[0]!.joints['thumb-tip'] = thumb + hands[0]!.joints['index-finger-tip'] = index + hands[0]!.add(thumb, index) + hands[0]!.updateMatrixWorld(true) + + system.update(0, 1 / 60) + expect(runtime).toMatchObject({ + pinching: true, + grabbedBy: 'pinch' + }) + + thumb.position.z = -0.4 + index.position.z = -0.4 + hands[0]!.updateMatrixWorld(true) + system.update(16, 1 / 60) + expect(runtime.grabIntent).toBe('depth') + expect(group.position.z).toBeGreaterThan(-1) + + thumb.position.z = -0.3 + index.position.z = -0.3 + thumb.position.x += 0.18 + index.position.x += 0.18 + hands[0]!.updateMatrixWorld(true) + system.update(32, 1 / 60) + expect(group.position.z).toBeGreaterThanOrEqual(-0.65) + expect(group.position.x).toBeCloseTo(0.334, 2) + + thumb.position.x = 0.08 + hands[0]!.updateMatrixWorld(true) + system.update(48, 1 / 60) + expect(runtime).toMatchObject({ + pinching: false, + grabbedBy: null + }) + expect(callbacks.onPanelMoved).toHaveBeenCalledTimes(1) + expect(callbacks.onPanelMoved).toHaveBeenCalledWith( + 'panel-1', + expect.objectContaining({ z: expect.closeTo(group.position.z) }) + ) + system.dispose() + }) + it('cancels a no-op hidden grab before clearing interaction ownership', () => { const hidden = createPanelDouble('hidden') const visible = createPanelDouble('visible') @@ -928,6 +1544,55 @@ describe('panel interaction model', () => { system.dispose() }) + it('accepts fingertip contact on the lowered scroll-up control', () => { + const { panel, scrollUpHit } = createPanelDouble() + scrollUpHit.visible = true + scrollUpHit.position.set(0, 0.25, 0.045) + const { system, hands, callbacks } = createInteractionHarness( + new Map([['panel-1', panel]]) + ) + const internals = system as unknown as { + sources: Array<{ + inputSource: XRInputSource | null + hand: XRHandSpace + handScrollPanelId: string | null + handScrollDirection: number + }> + updateHandPaneContact: ( + runtime: unknown, + now: number, + deltaSeconds: number + ) => void + } + const runtime = internals.sources[0]! + runtime.inputSource = { + handedness: 'right', + hand: {}, + targetRayMode: 'tracked-pointer' + } as unknown as XRInputSource + const index = Object.assign(new Group(), { + jointRadius: 0.01 + }) as unknown as XRJointSpace + index.position.set(0, 0.25, -0.91) + index.visible = true + hands[0]!.visible = true + hands[0]!.joints['index-finger-tip'] = index + hands[0]!.add(index) + hands[0]!.updateMatrixWorld(true) + + internals.updateHandPaneContact(runtime, 0, 0.1) + expect(callbacks.onScroll).toHaveBeenCalledWith( + 'panel-1', + expect.closeTo(-0.14), + 8 + ) + expect(runtime).toMatchObject({ + handScrollPanelId: 'panel-1', + handScrollDirection: -1 + }) + system.dispose() + }) + it('removes input listeners and disposes fallback geometry on teardown', () => { const targetRays = [new Group(), new Group()] const grips = [new Group(), new Group()] diff --git a/packages/webxr/test/panel-layout.test.ts b/packages/webxr/test/panel-layout.test.ts index ff390e8..e4ea572 100644 --- a/packages/webxr/test/panel-layout.test.ts +++ b/packages/webxr/test/panel-layout.test.ts @@ -56,7 +56,9 @@ describe('panel slot allocation', () => { displaced: { id: 'panel-0', slot: 4 - } + }, + overflowed: null, + position: null }) const placements = allocatePanelSlots([ @@ -94,4 +96,45 @@ describe('panel slot allocation', () => { expect(placements.find(placement => placement.id === 'automatic')) .toMatchObject({ slot: 0 }) }) + + it('avoids manually positioned panes when another screen anchor is clear', () => { + const manual = { + ...snapshot('manual'), + userMoved: true, + slot: 0, + position: { x: -1.1, y: 1.35, z: 0.2 } + } + expect(assignNewPanelToFrontSlot([manual])).toMatchObject({ + slot: 1, + displaced: null, + overflowed: null, + position: null + }) + }) + + it('places a necessarily overlapping pane in front of manual panes', () => { + const manual = { + ...snapshot('manual'), + userMoved: true, + slot: 0, + position: { x: 0, y: 1.775, z: 0.5 } + } + const assignment = assignNewPanelToFrontSlot([manual])! + expect(assignment.slot).toBe(0) + expect(assignment.position).toMatchObject({ x: -1.1, y: 1.35 }) + expect(assignment.position!.z).toBeGreaterThan(manual.position.z) + }) + + it('keeps the newest pane in front when every automatic slot is full', () => { + const panels = Array.from({ length: 8 }, (_, index) => ({ + ...snapshot(`panel-${index}`), + slot: index + })) + expect(assignNewPanelToFrontSlot(panels)).toEqual({ + slot: 0, + displaced: { id: 'panel-0', slot: 4 }, + overflowed: { id: 'panel-4' }, + position: null + }) + }) }) diff --git a/packages/webxr/test/panel-model.test.ts b/packages/webxr/test/panel-model.test.ts index 4bd9880..c2a2d57 100644 --- a/packages/webxr/test/panel-model.test.ts +++ b/packages/webxr/test/panel-model.test.ts @@ -4,6 +4,7 @@ import { SpatialPanelModel, type SpatialPanelInput } from '../src/panel-model' +import { allocatePanelSlots } from '../src/panel-layout' const panel = ( input: Partial = {} @@ -237,6 +238,32 @@ describe('spatial panel model', () => { }) }) + it('keeps the newest panel visible in front at full capacity', () => { + const model = new SpatialPanelModel() + for (let index = 0; index < 9; index += 1) { + model.upsert(panel({ + id: `command:${index}`, + title: `Command ${index}` + }), index) + } + + const snapshots = model.snapshots() + expect(snapshots.find(snapshot => snapshot.id === 'command:8')?.slot) + .toBe(0) + expect(snapshots.find(snapshot => snapshot.id === 'command:4')?.slot) + .toBe(4) + expect(snapshots.find(snapshot => snapshot.id === 'command:0')?.slot) + .toBe(null) + const placements = allocatePanelSlots( + snapshots, + { x: 0, y: 0, z: -2.4 } + ) + expect(placements.find(placement => placement.id === 'command:8')) + .toMatchObject({ slot: 0, overflow: false }) + expect(placements.find(placement => placement.id === 'command:0')) + .toMatchObject({ overflow: true }) + }) + it('bursts a manually dismissed panel and never resurrects it', () => { const model = new SpatialPanelModel() model.upsert(panel(), 0) diff --git a/packages/webxr/test/panel-view.test.ts b/packages/webxr/test/panel-view.test.ts index 0371875..8404028 100644 --- a/packages/webxr/test/panel-view.test.ts +++ b/packages/webxr/test/panel-view.test.ts @@ -9,6 +9,7 @@ import { resolvePanelHeight, resolvePanelControlLayout, resolvePanelInteractionLayout, + resolvePanelOverflowControlLayout, resolvePanelSlotTransition, resolvePanelViewportStart, resolvePanelVisualState @@ -48,6 +49,21 @@ describe('spatial panel visual states', () => { expect(layout.move).toMatchObject({ width: 1.55, height: 0.92, y: 0 }) }) + it('keeps scroll arrows inside the content area below the title bar', () => { + const interaction = resolvePanelInteractionLayout(1.55, 0.92) + const overflow = resolvePanelOverflowControlLayout(1.55, 0.92) + const titleBottom = interaction.titleBar.y + - (interaction.titleBar.height / 2) + const contentTop = interaction.content.y + + (interaction.content.height / 2) + const contentBottom = interaction.content.y + - (interaction.content.height / 2) + + expect(overflow.upY).toBeLessThan(contentTop) + expect(overflow.upY).toBeLessThan(titleBottom) + expect(overflow.downY).toBeGreaterThan(contentBottom) + }) + it('keeps interaction state out of the content render signature', () => { const content = { title: 'Run tests', @@ -113,6 +129,51 @@ describe('spatial panel visual states', () => { view.dispose() }) + it('keeps overflow hit targets available to controller rays', () => { + vi.stubGlobal('document', { + createElement: () => ({ + width: 0, + height: 0, + getContext: () => ({}) + }) + }) + vi.spyOn(CanvasTextSurface.prototype, 'render').mockReturnValue({ + totalLineCount: 8, + visibleLineCount: 2, + startLine: 2, + endLine: 4, + hasAbove: true, + hasBelow: true + }) + const view = new SpatialPanelView({ + id: 'panel-controller-scroll', + kind: 'command', + title: 'Controller scroll', + status: 'in-progress', + text: 'one\ntwo\nthree\nfour\nfive\nsix\nseven\neight', + retainedText: 'one\ntwo\nthree\nfour\nfive\nsix\nseven\neight', + truncated: false, + background: false, + phase: 'visible', + phaseStartedAt: 0, + scrollOffset: 2, + autoFollow: false, + userMoved: true, + position: null, + slot: 0, + fileTransitionStartedAt: 0 + }) + + expect(view.scrollUpHit.visible).toBe(false) + expect(view.scrollDownHit.visible).toBe(false) + view.setRayControlsVisible(true) + expect(view.scrollUpHit.visible).toBe(true) + expect(view.scrollDownHit.visible).toBe(true) + expect(view.scrollUpHit.parent?.visible).toBe(true) + expect(view.scrollDownHit.parent?.visible).toBe(true) + view.dispose() + }) + it('rerenders only when the effective clamped viewport line changes', () => { vi.stubGlobal('document', { createElement: () => ({ diff --git a/packages/webxr/test/passthrough-contrast.test.ts b/packages/webxr/test/passthrough-contrast.test.ts new file mode 100644 index 0000000..cf981c1 --- /dev/null +++ b/packages/webxr/test/passthrough-contrast.test.ts @@ -0,0 +1,69 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { NormalBlending } from 'three' +import { + PASSTHROUGH_DARK_FEATHER_STOPS, + PassthroughContrastView +} from '../src/passthrough-contrast' + +afterEach(() => { + vi.unstubAllGlobals() +}) + +describe('passthrough orb contrast', () => { + it('defines one monotonic neutral-dark feather ending fully transparent', () => { + expect(PASSTHROUGH_DARK_FEATHER_STOPS[0]?.[0]).toBe(0) + expect(PASSTHROUGH_DARK_FEATHER_STOPS.at(-1)).toEqual([ + 1, + 'rgba(5, 7, 9, 0)' + ]) + expect(PASSTHROUGH_DARK_FEATHER_STOPS.every( + ([offset], index, stops) => index === 0 || offset > stops[index - 1]![0] + )).toBe(true) + expect(PASSTHROUGH_DARK_FEATHER_STOPS.map(([, color]) => color).join(' ')) + .not.toMatch(/noise|dot|stipple|checker/i) + }) + + it('uses a smooth sprite only for alpha blend and leaves opaque mode unchanged', () => { + const addColorStop = vi.fn() + const context = { + createRadialGradient: vi.fn(() => ({ addColorStop })), + fillRect: vi.fn(), + fillStyle: '' + } + vi.stubGlobal('document', { + createElement: vi.fn(() => ({ + width: 0, + height: 0, + getContext: vi.fn(() => context) + })) + }) + const view = new PassthroughContrastView() + + expect(addColorStop.mock.calls).toEqual( + PASSTHROUGH_DARK_FEATHER_STOPS.map(stop => [...stop]) + ) + expect(view.group.visible).toBe(false) + expect(view.feather.name).toBe('alpha-passthrough-dark-feather') + expect(view.feather.userData).toMatchObject({ + contrast: 'dark-feather', + pattern: 'smooth-radial-gradient' + }) + expect(view.feather.material).toMatchObject({ + transparent: true, + blending: NormalBlending, + depthWrite: false, + opacity: 0.84 + }) + + view.setBlendMode('alpha-blend') + expect(view.group.visible).toBe(true) + expect(view.feather.visible).toBe(true) + expect(view.additiveOutline.visible).toBe(false) + view.setBlendMode('additive') + expect(view.feather.visible).toBe(false) + expect(view.additiveOutline.visible).toBe(true) + view.setBlendMode('opaque') + expect(view.group.visible).toBe(false) + view.dispose() + }) +}) diff --git a/packages/webxr/test/status-window-model.test.ts b/packages/webxr/test/status-window-model.test.ts index 556a66e..81d9dd7 100644 --- a/packages/webxr/test/status-window-model.test.ts +++ b/packages/webxr/test/status-window-model.test.ts @@ -12,6 +12,7 @@ import { resolveStatusContext, resolveStatusWindowScale, shouldShowStatusFallbackMenu, + StatusActionInteractionModel, StatusControllerArmModel, StatusGestureModel } from '../src/status-window-model' @@ -110,11 +111,33 @@ describe('XR status window model', () => { voiceState: 'resume-audio', reducedEffects: true })).toEqual([ - expect.objectContaining({ id: 'passthrough', available: false, state: 'Off' }), - expect.objectContaining({ id: 'recenter' }), - expect.objectContaining({ id: 'voice', label: 'Resume audio' }), - expect.objectContaining({ id: 'reduced-effects', state: 'On' }), - expect.objectContaining({ id: 'exit' }) + expect.objectContaining({ + id: 'passthrough', + available: false, + presentation: 'toggle', + checked: false + }), + expect.objectContaining({ + id: 'recenter', + presentation: 'button', + checked: null + }), + expect.objectContaining({ + id: 'voice', + label: 'Resume audio', + presentation: 'button', + checked: null + }), + expect.objectContaining({ + id: 'reduced-effects', + presentation: 'toggle', + checked: true + }), + expect.objectContaining({ + id: 'exit', + presentation: 'button', + checked: null + }) ]) }) @@ -193,21 +216,146 @@ describe('XR status window model', () => { tracked: true, controllerActive: false, wristHeightFromEyes: -0.2, - handBackFacingViewer: 0.7 + handBackFacingViewer: 0.7, + gazeAtHandDot: 0.95 } const closed = { open: false, invocation: null } expect(gesture.update({ ...posed, now: 0 }, closed)).toBe(null) - expect(gesture.update({ ...posed, now: 449 }, closed)).toBe(null) - expect(gesture.update({ ...posed, now: 450 }, closed)).toBe('open') - const lowered = { - ...posed, - wristHeightFromEyes: -0.6, - handBackFacingViewer: 0.1 - } + expect(gesture.update({ ...posed, now: 699 }, closed)).toBe(null) + expect(gesture.update({ ...posed, now: 700 }, closed)).toBe('open') const handOpen = { open: true, invocation: 'hand' as const } - expect(gesture.update({ ...lowered, now: 500 }, handOpen)).toBe(null) - expect(gesture.update({ ...lowered, now: 680 }, handOpen)).toBe('close') - expect(gesture.update({ ...posed, now: 1_000 }, closed)).toBe(null) + expect(gesture.update({ ...posed, now: 701 }, handOpen)).toBe(null) + expect(gesture.update({ ...posed, now: 881 }, handOpen)).toBe(null) + expect(gesture.update({ ...posed, now: 981, wristHeightFromEyes: -0.3 }, handOpen)).toBe(null) + expect(gesture.update({ ...posed, now: 1_081, wristHeightFromEyes: -0.42 }, handOpen)).toBe(null) + expect(gesture.update({ ...posed, now: 1_400, wristHeightFromEyes: -0.42 }, handOpen)).toBe(null) + expect(gesture.update({ ...posed, now: 1_401, wristHeightFromEyes: -0.42 }, handOpen)).toBe('close') + expect(gesture.update({ ...posed, now: 2_150 }, closed)).toBe(null) + }) + + it('keeps modest post-open motion inside the pose hysteresis', () => { + const gesture = new StatusGestureModel() + const open = { open: true, invocation: 'hand' as const } + const moved = { + tracked: true, + controllerActive: false, + wristHeightFromEyes: -0.3, + handBackFacingViewer: -0.2, + gazeAtHandDot: 0.95 + } + expect(gesture.update({ ...moved, now: 0 }, open)).toBe(null) + expect(gesture.update({ ...moved, now: 1_000 }, open)).toBe(null) + }) + + it('requires the viewer to keep looking toward the raised hand during reveal dwell', () => { + const gesture = new StatusGestureModel() + const closed = { open: false, invocation: null } + const sample = { + tracked: true, + controllerActive: false, + wristHeightFromEyes: -0.15, + handBackFacingViewer: 0.8, + gazeAtHandDot: 0.5 + } + expect(gesture.update({ ...sample, now: 0 }, closed)).toBe(null) + expect(gesture.update({ ...sample, now: 1_000 }, closed)).toBe(null) + expect(gesture.update({ ...sample, now: 1_001, gazeAtHandDot: 0.95 }, closed)).toBe(null) + expect(gesture.update({ ...sample, now: 1_700, gazeAtHandDot: 0.95 }, closed)).toBe(null) + expect(gesture.update({ ...sample, now: 1_701, gazeAtHandDot: 0.95 }, closed)).toBe('open') + }) + + it('dismisses only after the hand back is deliberately turned away', () => { + const gesture = new StatusGestureModel() + const open = { open: true, invocation: 'hand' as const } + const sample = { + tracked: true, + controllerActive: false, + wristHeightFromEyes: -0.15, + handBackFacingViewer: 0.8, + gazeAtHandDot: 0.95 + } + expect(gesture.update({ ...sample, now: 0 }, open)).toBe(null) + expect(gesture.update({ ...sample, now: 180 }, open)).toBe(null) + expect(gesture.update({ ...sample, now: 200, handBackFacingViewer: -0.4 }, open)).toBe(null) + expect(gesture.update({ ...sample, now: 759, handBackFacingViewer: -0.4 }, open)).toBe(null) + expect(gesture.update({ ...sample, now: 760, handBackFacingViewer: -0.4 }, open)).toBe('close') + }) + + it('cancels dismissal motion while the right hand is selecting', () => { + const gesture = new StatusGestureModel() + const open = { open: true, invocation: 'hand' as const } + const sample = { + tracked: true, + controllerActive: false, + wristHeightFromEyes: -0.15, + handBackFacingViewer: 0.8, + gazeAtHandDot: 0.95 + } + expect(gesture.update({ ...sample, now: 0 }, open)).toBe(null) + expect(gesture.update({ ...sample, now: 180 }, open)).toBe(null) + expect(gesture.update({ + ...sample, + now: 300, + wristHeightFromEyes: -0.4, + handBackFacingViewer: -0.5, + rightHandEngaged: true + }, open)).toBe(null) + expect(gesture.update({ + ...sample, + now: 1_000, + wristHeightFromEyes: -0.4, + handBackFacingViewer: -0.5, + rightHandEngaged: true + }, open)).toBe(null) + expect(gesture.update({ + ...sample, + now: 1_699, + wristHeightFromEyes: -0.4, + handBackFacingViewer: -0.5 + }, open)).toBe(null) + expect(gesture.update({ + ...sample, + now: 1_700, + wristHeightFromEyes: -0.4, + handBackFacingViewer: 0.8 + }, open)).toBe(null) + }) + + it('arms only after emergence and snapshots stale press/contact state', () => { + const interaction = new StatusActionInteractionModel() + interaction.updateWindow({ now: 0, open: true, fullyOpen: false }) + expect(interaction.phase).toBe('emerging') + expect(interaction.updatePress('right-controller', true)).toBe(false) + expect(interaction.updateContact('right-hand', 'passthrough')).toBe(null) + + interaction.updateWindow({ now: 160, open: true, fullyOpen: true }) + expect(interaction.phase).toBe('arming') + interaction.finishFrame(339) + expect(interaction.phase).toBe('arming') + interaction.finishFrame(340) + expect(interaction.phase).toBe('armed') + + expect(interaction.updatePress('right-controller', true)).toBe(false) + expect(interaction.updateContact('right-hand', 'passthrough')).toBe(null) + + expect(interaction.updatePress('right-controller', false)).toBe(false) + expect(interaction.updatePress('right-controller', true)).toBe(true) + expect(interaction.updatePress('right-controller', true)).toBe(false) + + expect(interaction.updateContact('right-hand', null)).toBe(null) + expect(interaction.updateContact('right-hand', 'passthrough')) + .toBe('passthrough') + expect(interaction.updateContact('right-hand', 'passthrough')).toBe(null) + }) + + it('resets action arming completely when the window closes', () => { + const interaction = new StatusActionInteractionModel() + interaction.updateWindow({ now: 0, open: true, fullyOpen: true }) + interaction.finishFrame(180) + expect(interaction.phase).toBe('armed') + interaction.updateWindow({ now: 181, open: false, fullyOpen: false }) + expect(interaction.phase).toBe('closed') + expect(interaction.updatePress('right-controller', true)).toBe(false) }) it('does not let the hand gesture close a controller-opened window', () => { @@ -217,7 +365,8 @@ describe('XR status window model', () => { tracked: false, controllerActive: true, wristHeightFromEyes: -1, - handBackFacingViewer: -1 + handBackFacingViewer: -1, + gazeAtHandDot: -1 }, { open: true, invocation: 'controller' })).toBe(null) }) @@ -234,17 +383,18 @@ describe('XR status window model', () => { expect(lost.update({ ...state, now: 1_300, tracked: false, gripHeightFromEyes: -1 })).toBe('close') }) - it('gives hand tracking loss a grace period before dismissal', () => { + it('keeps a hand-opened window available through tracking loss', () => { const gesture = new StatusGestureModel() const state = { open: true, invocation: 'hand' as const } const sample = { tracked: false, controllerActive: false, wristHeightFromEyes: -1, - handBackFacingViewer: -1 + handBackFacingViewer: -1, + gazeAtHandDot: -1 } expect(gesture.update({ ...sample, now: 0 }, state)).toBe(null) - expect(gesture.update({ ...sample, now: 299 }, state)).toBe(null) - expect(gesture.update({ ...sample, now: 300 }, state)).toBe('close') + expect(gesture.update({ ...sample, now: 500 }, state)).toBe(null) + expect(gesture.update({ ...sample, now: 5_000 }, state)).toBe(null) }) }) diff --git a/packages/webxr/test/status-window-placement.test.ts b/packages/webxr/test/status-window-placement.test.ts new file mode 100644 index 0000000..a0ba308 --- /dev/null +++ b/packages/webxr/test/status-window-placement.test.ts @@ -0,0 +1,138 @@ +import { describe, expect, it } from 'vitest' +import { Vector3 } from 'three' +import { + resolveHandStatusWindowAnchorPosition, + resolveStatusWindowAnchorPosition, + StatusWindowAnchorTracker, + STATUS_WINDOW_DISTANCE_METERS, + STATUS_WINDOW_WRIST_RISE_METERS +} from '../src/status-window-placement' + +describe('status window placement', () => { + it('keeps controller placement forty centimeters from the viewer', () => { + const viewer = new Vector3(0, 1.65, 0) + const wrist = new Vector3(-0.22, 1.42, -0.34) + const position = resolveStatusWindowAnchorPosition(wrist, viewer) + expect(position.distanceTo(viewer)).toBeCloseTo( + STATUS_WINDOW_DISTANCE_METERS + ) + expect(position.x).toBeLessThan(0) + expect(position.y).toBeCloseTo(1.65 + ( + (wrist.y + STATUS_WINDOW_WRIST_RISE_METERS - 1.65) + / new Vector3( + wrist.x, + wrist.y + STATUS_WINDOW_WRIST_RISE_METERS - 1.65, + wrist.z + ).length() + * STATUS_WINDOW_DISTANCE_METERS + )) + }) + + it('keeps hand placement attached to the physical wrist in all axes', () => { + const firstWrist = new Vector3(-0.18, 1.44, -0.34) + const secondWrist = new Vector3(-0.32, 1.5, -0.62) + const first = resolveHandStatusWindowAnchorPosition(firstWrist).clone() + const second = resolveHandStatusWindowAnchorPosition(secondWrist).clone() + expect(first).toEqual(new Vector3( + firstWrist.x, + firstWrist.y + STATUS_WINDOW_WRIST_RISE_METERS, + firstWrist.z + )) + expect(second).toEqual(new Vector3( + secondWrist.x, + secondWrist.y + STATUS_WINDOW_WRIST_RISE_METERS, + secondWrist.z + )) + expect(second.z - first.z).toBeCloseTo(secondWrist.z - firstWrist.z) + }) + + it('freezes the last stable pose during right-hand approach and tracking loss', () => { + const tracker = new StatusWindowAnchorTracker() + const first = tracker.update({ + wristPosition: new Vector3(-0.18, 1.44, -0.34), + selectionEngaged: false, + deltaSeconds: 1 / 60 + })!.clone() + const engaged = tracker.update({ + wristPosition: new Vector3(-0.28, 1.4, -0.3), + selectionEngaged: true, + deltaSeconds: 1 / 60 + })!.clone() + const occluded = tracker.update({ + wristPosition: null, + selectionEngaged: true, + deltaSeconds: 1 / 60 + })!.clone() + + expect(engaged).toEqual(first) + expect(occluded).toEqual(first) + }) + + it('rejects an implausible reacquisition jump but follows modest motion', () => { + const tracker = new StatusWindowAnchorTracker() + const first = tracker.update({ + wristPosition: new Vector3(-0.18, 1.44, -0.34), + selectionEngaged: false, + deltaSeconds: 1 / 60 + })!.clone() + const jumped = tracker.update({ + wristPosition: new Vector3(0.45, 0.8, -0.05), + selectionEngaged: false, + deltaSeconds: 1 / 60 + })!.clone() + const followed = tracker.update({ + wristPosition: new Vector3(-0.22, 1.43, -0.34), + selectionEngaged: false, + deltaSeconds: 1 / 30 + })!.clone() + + expect(jumped).toEqual(first) + expect(followed.x).toBeLessThan(first.x) + }) + + it('reacquires a stable wrist pose after rejecting the initial jump', () => { + const tracker = new StatusWindowAnchorTracker() + const first = tracker.update({ + wristPosition: new Vector3(-0.18, 1.44, -0.34), + selectionEngaged: false, + deltaSeconds: 1 / 60 + })!.clone() + const reacquiredWrist = new Vector3(0.2, 1.2, -0.5) + + for (let index = 0; index < 8; index += 1) { + tracker.update({ + wristPosition: reacquiredWrist, + selectionEngaged: false, + deltaSeconds: 1 / 60 + }) + } + const reacquired = tracker.update({ + wristPosition: reacquiredWrist, + selectionEngaged: false, + deltaSeconds: 1 / 60 + })!.clone() + + expect(reacquired.x).toBeGreaterThan(first.x) + expect(reacquired.y).toBeLessThan(first.y) + }) + + it('does not reacquire while the wrist keeps drifting', () => { + const tracker = new StatusWindowAnchorTracker() + const first = tracker.update({ + wristPosition: new Vector3(-0.18, 1.44, -0.34), + selectionEngaged: false, + deltaSeconds: 1 / 60 + })!.clone() + + let tracked = first + for (let index = 0; index < 12; index += 1) { + tracked = tracker.update({ + wristPosition: new Vector3(0.2 + index * 0.03, 1.2, -0.5), + selectionEngaged: false, + deltaSeconds: 1 / 60 + })!.clone() + } + + expect(tracked).toEqual(first) + }) +}) diff --git a/packages/webxr/test/status-window-view.test.ts b/packages/webxr/test/status-window-view.test.ts new file mode 100644 index 0000000..6ec823a --- /dev/null +++ b/packages/webxr/test/status-window-view.test.ts @@ -0,0 +1,84 @@ +import { describe, expect, it, vi } from 'vitest' +import { + drawStatusToggle, + drawStatusWindowFrame, + STATUS_WINDOW_ACTION_APPEARANCE, + STATUS_WINDOW_APPEARANCE, + STATUS_WINDOW_CORNER_RADIUS_PIXELS, + STATUS_WINDOW_HEIGHT_METERS, + STATUS_WINDOW_HIT_DEPTH_METERS, + STATUS_WINDOW_TOGGLE_APPEARANCE, + STATUS_WINDOW_WIDTH_METERS +} from '../src/status-window-view' + +describe('status window view', () => { + it('uses the requested one-third physical size', () => { + expect(STATUS_WINDOW_WIDTH_METERS).toBeCloseTo(0.72 / 3) + expect(STATUS_WINDOW_HEIGHT_METERS).toBeCloseTo(0.96 / 3) + expect(STATUS_WINDOW_HIT_DEPTH_METERS).toBeLessThan(0.01) + }) + + it('draws one rounded saturated lime frame with a soft glow', () => { + const context = { + beginPath: vi.fn(), + roundRect: vi.fn(), + fill: vi.fn(), + stroke: vi.fn(), + fillStyle: '', + strokeStyle: '', + lineWidth: 0, + shadowColor: '', + shadowBlur: 0 + } + drawStatusWindowFrame(context) + expect(context.roundRect).toHaveBeenCalledWith( + 12, + 12, + 696, + 936, + STATUS_WINDOW_CORNER_RADIUS_PIXELS + ) + expect(context.fillStyle).toBe(STATUS_WINDOW_APPEARANCE.background) + expect(context.strokeStyle).toBe(STATUS_WINDOW_APPEARANCE.border) + expect(context.shadowColor).toBe(STATUS_WINDOW_APPEARANCE.glow) + expect(context.lineWidth).toBe(6) + expect(context.fill).toHaveBeenCalledTimes(1) + expect(context.stroke).toHaveBeenCalledTimes(1) + expect(context.shadowBlur).toBe(0) + expect(STATUS_WINDOW_APPEARANCE.accent).toBe('#c8ff4d') + }) + + it('uses a distinctly brighter lime fill for direct-touch feedback', () => { + expect(STATUS_WINDOW_ACTION_APPEARANCE.pressedFill) + .not.toBe(STATUS_WINDOW_ACTION_APPEARANCE.idleFill) + expect(STATUS_WINDOW_ACTION_APPEARANCE.pressedFill) + .toContain('0.42') + expect(STATUS_WINDOW_ACTION_APPEARANCE.pressedBorder) + .toContain('0.96') + }) + + it('moves a real toggle knob between off and on states', () => { + const context = { + beginPath: vi.fn(), + roundRect: vi.fn(), + fill: vi.fn(), + fillStyle: '' + } + drawStatusToggle(context, { + checked: false, + available: true, + centerY: 620 + }) + const offKnobX = context.roundRect.mock.calls[1]?.[0] as number + expect(context.fillStyle).toBe(STATUS_WINDOW_TOGGLE_APPEARANCE.knob) + + context.roundRect.mockClear() + drawStatusToggle(context, { + checked: true, + available: true, + centerY: 620 + }) + const onKnobX = context.roundRect.mock.calls[1]?.[0] as number + expect(onKnobX).toBeGreaterThan(offKnobX) + }) +}) diff --git a/packages/webxr/test/transcript-bubble-view.test.ts b/packages/webxr/test/transcript-bubble-view.test.ts index 2b4c085..ccc9863 100644 --- a/packages/webxr/test/transcript-bubble-view.test.ts +++ b/packages/webxr/test/transcript-bubble-view.test.ts @@ -1,10 +1,18 @@ import { describe, expect, it } from 'vitest' import { resolveTranscriptBubbleScale, - shouldRenderTranscriptTexture + shouldRenderTranscriptTexture, + TRANSCRIPT_BUBBLE_APPEARANCE } from '../src/transcript-bubble-view' describe('assistant transcript bubble animation', () => { + it('uses a sky-blue translucent surface for passthrough contrast', () => { + expect(TRANSCRIPT_BUBBLE_APPEARANCE.background) + .toBe('rgba(12, 70, 96, 0.62)') + expect(TRANSCRIPT_BUBBLE_APPEARANCE.border) + .toBe('rgba(104, 225, 255, 0.96)') + expect(TRANSCRIPT_BUBBLE_APPEARANCE.text).toBe('#d9f8ff') + }) it('scales in and out over a quarter second', () => { expect(resolveTranscriptBubbleScale('hidden', 0)).toBe(0) expect(resolveTranscriptBubbleScale('appearing', 0)).toBe(0) diff --git a/packages/webxr/test/voice-runtime.test.ts b/packages/webxr/test/voice-runtime.test.ts index 40270c1..a657963 100644 --- a/packages/webxr/test/voice-runtime.test.ts +++ b/packages/webxr/test/voice-runtime.test.ts @@ -84,6 +84,7 @@ vi.mock('@codori/client/shared/realtime', () => ({ import { resolveImmersiveVoiceActivity, + voiceSessionActive, VoiceRuntime } from '../src/voice-runtime' @@ -166,6 +167,23 @@ describe('voice runtime', () => { })).toBe('speaking') }) + it.each([ + 'requesting-permission', + 'creating-offer', + 'starting', + 'connected', + 'stopping' + ] as const)('treats %s as an active voice session', (state) => { + expect(voiceSessionActive(snapshot(state))).toBe(true) + }) + + it.each([ + 'idle', + 'error' + ] as const)('allows orb voice start while %s', (state) => { + expect(voiceSessionActive(snapshot(state))).toBe(false) + }) + it('enables the microphone after an idle capability refresh connects', async () => { const runtime = new VoiceRuntime({ client: createClient(), diff --git a/packages/webxr/test/xr-capability.test.ts b/packages/webxr/test/xr-capability.test.ts index 4d3d014..5b0b94f 100644 --- a/packages/webxr/test/xr-capability.test.ts +++ b/packages/webxr/test/xr-capability.test.ts @@ -140,7 +140,11 @@ describe('immersive WebXR capability', () => { vrSupported: true, mode: 'immersive-ar', environmentBlendMode: 'alpha-blend' - })).toMatchObject({ supported: true, active: true, contrast: 'dither' }) + })).toMatchObject({ + supported: true, + active: true, + contrast: 'dark-feather' + }) expect(resolvePassthroughAvailability({ arSupported: true, vrSupported: true,