Facebook announced a new API that will let developers incorporate video from the Quest 2’s sensors into their games and applications, creating a mixed reality experience. With the Passthrough API, developers will not only be able to mix the black-and-white images from the headset’s sensors into their experiences, but they’ll also be able to customize how it ends up looking to the player, apply effects, and even have the real world show up on specific surfaces.
Facebook’s announcement includes examples of how the API could improve productivity software by allowing integration of real-life keyboards and desks, and how games could include your real-life furniture for enemies to hide behind. It’s also easy to imagine other fun experiences that mix your real environment with a virtual world — how about Portal-esque puzzles that you solve in your own house, or the ability to see your walls covered in virtual paint?
Facebook told UploadVR that the API was currently only for the Quest 2 when asked about compatibility with the first-generation Quest. In the announcement post, Facebook says the image processing for the API is done on-device and that applications using it won’t be able to see or store the images from the Quest 2’s sensors.
Facebook is adding the experimental version of the API to its upcoming developer tools, and it’ll be available first to programmers using Unity to build their software. Facebook does promise that the API will be getting support for other development platforms in the future.
The Quest, Quest 2, and Rift S headsets already have a version of the Passthrough technology, which lets users get a quick look at what’s happening around them in the real world. Facebook also lets you set Passthrough as your virtual environment, giving you a version of your real environment (albeit in black and white) that you can navigate the Oculus UI in. It even showed off that ability in a video about its Infinite Office feature.
[embedded content]
The ability to integrate your real environment into a virtual world has long been one of the more exciting promises of headsets with integrated cameras. Microsoft even invoked the idea when it named its VR platform Windows Mixed Reality (regardless of how much that’s actually been delivered on).
The news of Oculus’ API doesn’t mean that you’ll be able to pick up your headset and have a mixed reality experience today, however. In its announcement, Facebook says it’s looking to let developers ship software using Passthrough to players “later this year.” Still, this API’s availability to developers is exciting, as it should allow for new types of experiences on the Oculus platform.
