Android XR to Enable Camera Access Similar to Smartphones

Passthrough camera access has become a buzzworthy topic in the XR community lately. We’re clear on where Meta, Apple, and Pico stand, but there’s a lot of curiosity about Google’s direction with Android XR. After having an insightful conversation with a contact at Google, I can reveal that their approach will mirror what we’re familiar with on smartphones. Stick with me for the details!

The Camera Access Dilemma

Let’s rewind for a moment for those of you wondering what’s at play here. The newest standalone VR headsets, which are essentially MR headsets, bring us a vivid RGB passthrough view thanks to the front cameras. This allows for awesome mixed reality experiences, including games like Cubism, Starship Home, and Pencil.

The operating system taps into camera frames to display the passthrough, and as developers, we wish we had access too. Imagine the potential: AI and computer vision could greatly enhance user experiences by interacting with their surroundings. In one of my previous articles, I argued that camera access is vital for true mixed reality. I managed to prototype an AI+MR app, using a workaround on the Quest, that assists with interior design. Such innovation wouldn’t be feasible without access.

Of course, this is where we hit the privacy concerns wall. Unscrupulous developers could exploit camera access to gather sensitive info, like images of IDs or credit cards lying around. It’s also a worry that images of people’s faces or bodies could be misused.

Balancing user privacy with unleashing mixed reality’s potential is indeed a sensitive matter.

How XR Companies Are Handling It

There was a time when full camera access was freely given in early XR days. If you’ve been following my journey, you might recall our experiments with the Vive Focus back in 2019 — working on diminished reality, sound reactivity, and more.

As mixed reality gained traction, companies became cautious, sacrificing access to camera frames due to privacy concerns. Names like Meta, Pico, HTC, and Apple all put up barriers, shutting developers out.

This became the norm, until XR developers recognized the pressing need for this feature and started lobbying for it. Key voices like Cix Liv, Michael Gschwandtner, and myself pushed for a controlled access framework, mirroring camera permissions on smartphones.

This advocacy paid off; Meta, for instance, committed to releasing a "Passthrough API" this year. But what are Google’s plans for Android XR?

Android XR: Treating the Headset Like a Phone

With Android dominating global smartphone OSs, any Android app can ask a user’s permission to access camera streams. If users agree, developers specify which camera (like the back camera) they want to use. Google aims for similar compatibility with Android XR, enhancing the transition for app developers.

My efforts got confirmed after an extensive email exchange with a Google spokesperson on camera access for Android XR. Here’s the crux of what she said:

Developers can leverage existing camera frames for XR applications with user approval, parallel to phone permissions. Our documentation delves into other permissions apps can request, like accessing world-facing or selfie cameras. Both these streams can be accessed using Android Camera APIs, notably Camera2 and CameraX.

Thus, Android developers can use familiar tools, like CameraX, to manage camera streams on XR devices. This similarity makes it seamless on headsets and glasses, and it’s amazing news!

For the rear camera, however, developers get a reconstructed avatar stream akin to Apple’s Vision Pro. This consistency ensures Android XR’s behavior mirrors that on phones: the rear camera views the world in front, while the selfie camera offers the user’s avatar.

Google’s approach is clever, ensuring Android apps can run seamlessly on XR devices. Maintaining consistent permission requests across phones and headsets is something I’m absolutely behind.

A Little Caveat About Android XR

Keep in mind, Android XR is still in preview mode and hasn’t been officially launched on any headsets. This means things could change before the official release. While I doubt any drastic shifts will occur, it’s wise to consider the possibility.

The Opening Up of Camera Access

With Google and Meta paving the way for camera access, other companies are likely to follow. The upcoming year looks promising for mixed reality, unlocking its possibilities like never before. I’m excited to see what this means for the developer community’s creativity!

(Header image based on a Samsung image)

(Note: This blog includes advertisements and affiliate links to support its operation. Clicking on them earns me a small commission. You can read the full disclosure here.)

Remember to share this with fellow innovators!

Related