Interacting live with an iPhone-captured face in your Looking Glass Portrait

Hi everyone! I'm Albert and I'm a UX designer here at Looking Glass Factory. Here's something fun I hacked together with Looking Glass Portrait in the last few weeks:

What you're looking at here is a Looking Glass Portrait with a live stream from iPhone's TrueDepth camera using the Record3D app streaming functionality.  This live stream is being manipulated by a virtual hand controlled by my real hand through a Leap Motion controller.  Here's a photo of the setup, with a diagram below illustrating how the tech is hooked up.

This exploration was inspired long-time Looking Glass friend @asidys230.  He blew our minds with this tweet back in March earlier this year, showing off what he could accomplish with his Looking Glass 8.9" display.  When we saw this we knew we needed to do some investigation as well:

Getting the iPhone TrueDepth camera stream

The TrueDepth camera is what iPhone uses for Face ID and Animoji.  It drives these features with 3D scanning technology.

Fortunately, this 3D data is made available to Unity through an app called Record3D – a feature we just unlocked with our $2M stretch goal.  Among the many neat features this app provides, the paid version allows you to connect your iPhone* to your computer and use it like a 3D webcam to stream 3D data live to Unity.  The developer also provides an example Unity project, which uses the two textures (RGB & Depth) to drive a Visual Effect in VFX Graph.

(* this feature is only available for iPhone X, 11 and 12.)

Adding Leap Motion Control

The Leap Motion Controller is a small computer peripheral that tracks hands in 3D space.  This controller is one of those technologies we've used a lot for demos because it unlocks easy to understand 3D interaction.

In this case, the Leap Motion controller allows the user to interact with their own face, distorting the data streaming from Record3D.

Live Streaming to the Looking Glass Portrait

For this demo, Looking Glass Portrait is in what we call "Desktop Mode".  Unlike "Standalone Mode", where Looking Glass Portrait is plugged into only a power source and shows what is in on board memory, "Desktop Mode" means that the device is directly connected to a computer that is powerful enough to render the graphics in real-time. "Desktop Mode" functions almost identically to how the Looking Glass 8K, 15.6", and legacy 8.9" displays work: as an external 3D monitor to your computer setup.

In order to communicate with the device, Unity is loaded up with the HoloPlay Unity Plugin.  Unfortunately, the mainline version of this software does NOT support VFX in any flavor.  We do, however, have an old experimental version of the plugin, which supports VFX through LWRP.

Hooking (Hacking) it Together

So, to get this working:

• I cloned an old version of Record3D for Unity (the latest version was incompatible with our experimental Unity Plugin)
• Added the Leap Motion Core module
• Removed HRDP, added LRWP and VFX Preview Package
• Added the experimental LWRP HoloPlay Unity SDK

Then with this project I added some scripts and modified the VFX graphs to hook all these things up together.

Interaction Logic

In this demo, when the program saves a "pinch" event when it detects that the thumb and index finger are close together.  A vector is then generated that describes the direction and magnitude that the hand has dragged.  This vector is fed into VFX graph, which it uses to distort the image of the face.  Bigger drags mean bigger distortions.

Releasing the pinch creates an animation, bouncing the face back and forth.

Lastly, if pressing the mouse button toggles whether the distortions stick in place, or if they immediately rebound when the pinch is released.

A glimpse of what's to come

At Looking Glass Factory, we get to experiment and patch together a lot of amazing technologies to make novel experiences.  To add to this, we're really fortunate in that our community is filled with cutting-edge technologists who seem to be able to always challenge what we thought was possible, even with our tech.

Other Record3D Features: iPhone Video Recording

We just unlocked a $2M stretch goal to provide a license to Record3D to our backers on Kickstarter who purchase a Looking Glass Portrait.  We've started a conversation with the founder and lead developer of Record3D and are in the final points of finalizing the details, but it looks like Record3D should be able to record video in a file format that is compatible with HoloPlay Studio – our application for creating, editing and organizing holograms in Looking Glass Portrait.

Here's an example that Shawn, our co-founder and CEO, recorded as he as his kids were decorating their Christmas tree. We're all super excited that a lot of you will be able to start experimenting with holographic video next year too.

To chat more about experiments we're working on, join our team and community on Discord where 700+ community members chat all things 3D and holograms.

To pre-order Looking Glass Portrait, head on over to our Kickstarter page which will be live until January 14, 2021.