Hz Unity Passthrough Camera Access

Meta Quest Passthrough Camera Access (PCA)

Building on top of the Android Camera2 API,

Passthrough Camera Access provides access to the forward-facing RGB cameras on

Quest 3 and Quest 3S for the purpose of supporting Computer Vision and Machine

Learning. This API is distinct from the

Media Projection API,

which supports casting from the user's POV (including the UI) and should be used

if your purpose is to represent what the user is seeing. Use the Passthrough

Camera API to add application-specific computer-vision capabilities that extend

the understanding of the user's environment and actions beyond what is provided

by the Quest Scene API.

Use Cases

This API provides an unobstructed view using the forward-facing RGB cameras and

can be integrated with ML/CV pipelines. Common use cases:

• Specially-trained ML/CV models that identify specific objects the

application can interact with (e.g. fitness equipment like dumbbells,

industrial equipment like audio mixers).

• ML/CV assistants or guides for experiences. For instance, a museum tour

where looking at a painting surfaces information about its history, artist,

and style using off-device LLMs.

• Feedback for training or special-interest applications — after asking the

user to perform a task, the app can detect whether it was done correctly by

interpreting changes to the environment (e.g. writing on a whiteboard).

• Design improvements. By interpreting the lighting or using the camera image

to modify a texture, much more realistic design effects can be achieved.

General Prerequisites

• Horizon OS v74 or later.
• Quest 3 or Quest 3S (Quest Pro and earlier are not supported).
• Either permission android.permission.CAMERA or

horizonos.permission.HEADSET_CAMERA. CAMERA grants access to both the

passthrough and avatar cameras; HEADSET_CAMERA grants access only to the

passthrough camera.

• The Passthrough feature must be enabled to access the Passthrough

Camera API.

Passthrough Camera Using Android Camera2

Camera Access on Quest is implemented on top of

Android's Camera2 API

within Horizon OS. Starting from Horizon OS v74, Camera2 and its Unity

extension are available on Quest headsets (Horizon OS v83 added an Unreal

extension). On Quest 3 and Quest 3S developers have access to the left and right

cameras on the face of the HMD.

The Android Camera2 API provides:

• Image capture: capture camera data for advanced processing.
• Camera metadata: query the API for hardware capabilities and

configuration information.

• Multi-camera support: access and control multiple cameras.

Known Issues

• After the first install of an app using the Camera API in v74 after flashing

a new OS, the RGB channels might flip causing color distortion (e.g. flesh

tones appear bluish). Reboot the device to fix.

• Passthrough Camera API is not supported in XR Simulator.
• The passthrough camera texture captures a rectangular area (1280×960)

smaller than what the user sees. The 1280×1280 resolution added in v83

expands the vertical field of view, but still does not cover the entire

passthrough view.

• Restrictions applied by a parent to restrict Teen and Youth accounts (e.g.

accessing the Passthrough Camera API) are not applied if the application is

installed through the Meta Quest Developer Hub.

Best Practices

• Ensure user privacy. Camera image data is considered Device User Data and

is covered by the Developer Data Use Policy. Adhere to this policy fully.

• Avoid costly processing on device. On-device processing of camera images

can enable compelling use cases, but keep experiences comfortable by

maintaining a high framerate.

References

Detailed, task-specific guidance for projecting and placing PCA imagery in world

space lives under references/:

• references/principal-point-offset.md —

Correctly project the camera image onto a flat world-space surface

(frustum-slice quad, FOV visualizer, image-plane overlay) using the optical

axis as the plane normal. Read this when corner rays from ViewportPointToRay

produce a non-rectangular quad, a camera-aligned quad drifts off-center, or

2 * d * tan(fov / 2) gives the wrong size on Phoenix/Stanley but worked on

Quest 3.

• references/detection-bounding-boxes.md —

Draw precise 3D bounding boxes around real-world objects detected by an ML

model. Read this when placing 2D ML detections (YOLO, SSD, DETR, custom CNN)

into world space, building a camera-facing billboard quad from a 2D detection,

or wiring PassthroughCameraAccess + EnvironmentRaycastManager for object

localization.