With the rise of Mixed Reality headsets, camera access has become one of the most desired features for developers. In a recent interview, Meta’s CTO hinted at the possibility of enabling this feature in the future.
If you own a smartphone, you may know that its cameras are already accessible to developers. Of course, when you open an app that uses the camera, you will be asked for permissions, so developers can’t access it whenever they want. This is a good thing because people are naturally cautious about camera access due to privacy concerns. However, this is a problem that has been largely solved on smartphones.
Smartphones have their version of Augmented Reality, but Mixed Reality headsets (and future glasses) offer a superior experience. You can interact with virtual content using your hands or controllers and view it in full 6-DoF within your space. This is both fun and promising. However, developers currently cannot utilize the full potential of Mixed Reality due to the lack of camera access.
Currently, if you want to build a Mixed Reality app on the Meta Quest, there is little you can do to make your app dynamically react to the environment. You can’t see what the user sees. The Quest 3 has a room scanning feature, where the user manually scans the space, and the headset builds a mesh. However, this doesn’t fully solve the problem, as it doesn’t allow for features like object tracking.
In an interview between Matthew Ball and Andrew Bosworth (Meta’s CTO), Bosworth hinted that camera access might eventually come to Quest headsets. He expressed excitement about the possibilities this feature would unlock for developers but noted that Meta has taken a conservative stance on camera access due to privacy concerns. Given Meta’s history with privacy scandals, this caution is understandable. However, as the Mixed Reality industry grows and with Apple already allowing camera access for enterprise developers (suggesting it may eventually be available to regular developers), Meta might be compelled to follow suit.
Below are several examples of Mixed Reality/Augmented Reality apps that utilize camera access. Some are mere ideas, while others are finished projects that use (sometimes awkward) workarounds to function without direct camera access. Enabling camera access could make these apps much more user-friendly.
The winner of the Meta AR Hackathon – “Flaivor”
Flaivor is an AI-powered cooking assistant that, based on images of items in your fridge (or any other photo of available food), gives you recipes that include the ingredients you photographed. It also has a feature that lets you set timers and an AI assistant that provides help and guidance.
Did you catch it? Yes, this app requires you to take a picture with your headset. This is a clever (but not ideal) workaround to the lack of camera access, which would simplify the experience significantly. Imagine just looking at your fridge and focusing on specific ingredients that a single photo might miss—this would be possible with camera access.
Spatial Vacuuming!
This concept app is designed for fun but offers an interesting idea to make chores more enjoyable and less tedious. It includes features like earning coins at each spot and using different floor colors to indicate which parts haven’t been cleaned yet, as shown in the video. The creator achieved this by mounting the Meta Touch Pro controller, briefly visible in the video.
With camera access, it could be possible to track the vacuum cleaner without the need for any controllers. Since floor and spatial mesh data are already available to developers, camera access is the only missing component.
Quest 3 Mixed Reality app – “Pencil”
This app is particularly interesting, and I had a chance to try it myself. I made a drawing with it! It was… bad, but hey, it looked much better when I used this app compared to when I did it without. (And no, I’m not going to show you the drawing.)
This app teaches you drawing, essentially, as demonstrated in the video on Twitter. What it does is simple: it tracks an A4 paper sheet with a controller on top and adds a layer to help the user align it with the real piece of paper. With camera access, developers could actually track the paper itself, which would make the process a lot easier. Currently, a controller can move, causing the virtual sheet of paper to move with it, potentially misaligning the real and virtual drawings. Camera access would eliminate this issue.
“World Skins” – Transform your world with AI and AR
What’s intriguing about this example is how it utilizes AI and camera access to transform the entire world around you. While this is currently just a concept, it’s a very enjoyable one. We could discuss potential risks if one isn’t careful enough, but what’s more significant is the sheer potential to achieve something like this. For a safer but equally impressive experience, imagine applying the same concept within your own home: transforming it to resemble a bamboo-built jungle or a spaceship visible through your windows.
What’s the benefit of using Mixed Reality over VR? Well, one advantage could be the ability to see your furniture or people moving around you. Another benefit is the potential to use it outdoors, such as in your garden—an area that remains safe from hazards like cars but could be significantly enhanced through the use of AI.
AI Assistant that sees what you see
Large Language Models are improving rapidly. For instance, the one demonstrated in GPT-4o can “understand” your surroundings and context, thereby providing highly useful assistance when needed.
If Mixed Reality headsets and AR Glasses were granted camera access for developers, they could utilize their trained models for assistance. For example, if you encounter a problem with a tap and need a plumber’s help, you could potentially handle the issue independently. How? Simply look at the installation and ask the AI for assistance. With camera access, the AI would have instant access to what you see, greatly enhancing its ability to understand the situation and provide effective guidance.
Examples like these are endless, and many of them have not even been imagined yet. They could become reality if developers had access to headset cameras.
Meta’s Fascinating Reality Lab AR Research Project
This is a very cool demo from Meta Reality Labs. It allows the user to manipulate reality by not only adding virtual elements but also altering existing ones. For instance, you can move objects in the real world, and the headset will approximate the texture and colors behind them, creating a seamless virtual wall that replaces the real wall behind the moved object.
It is an approximation, so the generated wall isn’t identical to the real one, but it could be improved. The developer behind the project mentioned that careful scanning and pre-planning of each object were necessary for the demo. However, with camera access and advanced AI models, we can imagine rendering all of this in real-time, taking Mixed Reality to a whole new level.
Changing the world dynamically around you – Aldiffusion API
The prototype shown here is an AI-driven demonstration of possibilities. Not much to describe – you touch something in your room, and your hands (or body) change to match what you’ve touched. Is it a gimmick? Yes, certainly. Ideas like these could evolve into something bigger and much more useful though!
AR for Packaging
What if you enter a shop with your AR glasses in the future, and every product you see has a creative animation? That might be overwhelming, and I can’t imagine anyone enjoying seeing hundreds of products on the shelves moving. But if you could focus on a product, click your fingers, and see an animation showing how to use it or potential recipes with it (depending on whether it’s a consumable or something else), that would be amazing.
I can see the benefits of this. Of course, it would require camera access to work.
“Try-On” Augmented reality solution by Gucci
Another interesting AR use case that requires camera access is the “try on” feature. You can already do this on your phone, but it’s not well-supported. With a device capable of altering reality in a usable way already on your head, developers could expand features like this. I would certainly use it to see how shoes look on my feet before buying them, without needing to go to a store!
The same could apply when looking in the mirror, with the headset or glasses seeing your body. You could check if certain clothes fit well. There are already apps that sort of do this, but nothing can beat seeing the image directly on the glasses or headset.
There are many more examples like the ones above, but I believe they already give you a good idea of what’s possible. Meta and Apple are slowly moving towards allowing developers to access headset cameras. If this happens, it will mark the moment when innovative, creative, and useful Mixed Reality apps will truly emerge.
