Advanced Hologram Editing from Smartphone Portrait Mode Photos
What is a "Portrait mode" photo?
Many smartphones nowadays have intelligent camera systems that capture, in addition to color data, capture depth data as well. Here's an animation from Apple's 2016 Keynote explaining the depth-sensing technology that was integrated into iPhone Portrait photos.
Since that Keynote in 2016, practically all iPhones and many flagship Android phones have the ability to capture depth data that can be used to make holograms for the Looking Glass Portrait. To see whether your smartphone can capture depth data to make quality holograms, please view our list of devices with depth data support.
Hologram Editing Software ≠ Photo Editing Software
When we started experimenting with taking, editing, sharing, and presenting these depth photos in Looking Glass displays, we found we had a lot of needs that conventional 2D editing software wasn't able to satisfy. So we started experimenting and building tools to help us make the most this new type of data.
Below are a handful of features that really excite us about 3D capture technology. Some of these are being woven into the software we're releasing for the currently ongoing Looking Glass Portrait Kickstarter campaign, while others are a bit more speculative, but we're keenly interested in figuring out ways to get these features into the hands of Looking Glass Portrait owners.
The three major categories of hologram editing features we're most interested in exploring and sharing are:
- 3D Composition Tools
- Augmentation
- Patching & Repair
1. 3D Composition Tools
Cropping
When composing 2D photographs in post, you mostly have to worry about where the boundaries of the crop are (top / bottom / left / right).
In 3D it's a bit more complicated.
The most obvious additional tool needed is being able to crop against depth. In computer graphics, the near and far cropping planes are known as "clipping planes". So for us, we need a control for dragging the far clipping plane forward and back so that you can lop off a distracting thing in the background.
Synthetic Backgrounds
This produces an awkward effect where the subject now stands in a void of emptiness. To resolve this, our software offers ways for you to synthetically add in a 2D backdrop for your subject.
Controllable "Depthiness"
Another control we've found to be vital to hologram presentation is how "depthy" the hologram appears. Some holograms look better when made flatter, and others look better when the depth is exaggeratedly stretched. We've included depth sliders in our software allows for fine tune control on this property.
2. Augmentations
Perhaps the most exciting and magical editing tools that we've discovered or encountered are the ones that give hologram editors a sort of editorial power.
In the 2D world, a designer for a movie poster may use a photograph for the base layer of the poster. They then might use software to add text, integrate other images, or lighting / darken / sharpen / blur certain areas of the photo.
The challenge to designing software for this type of work (in both 2D and 3D) is not so much imagining the possibilities, but figuring out which ones to prioritize building first, and how to package these effects in ways that are both easy and empowering.
Here are some of the ones that we're most excited about.
3D Lighting and Shadows
Portrait mode photos combine conventional photo data with 3D geometries, so instead of relying on photo editing software to manually select portions of your photo to highlight / shadow, you can rely on the 3D geometry to produce realistic lighting effects on your holograms.
In our software, this manifests as environmental effects like the disco ball effect above, as well as static effects, like synthetically adding light sources in post.
Similarly, you can use the geometry to cast shadows. Shadows play a huge part in 3D perception, so being able to do this with simple controls with our software is very empowering.
GIFs and Annotations
Similar to how mobile photo editing software allows you to annotate your photos with stickers or text, we found that those annotations on holograms were just as powerful while remaining easy to manipulate. Though gifs are 2D, placing them inside a 3D context, though silly, really can add life to holograms. Text annotations can play a big role in contextualizing holographic information.
3. Patching & Repair
Though the smartphones are perhaps the most accessible 3D capture technologies on the planet, the quality of the capture isn't perfect. The depth data is optimized for the background blurring effect for 2D images, not for holograms. Depending on a variety of factors (lighting, subject, textures, etc...) sometimes the steps required to convert that data into a hologram produces depth artifacts.
At Looking Glass Factory we're working hard to account for these shortcomings. We don't have any footage to share on this yet, but some of the things we're exploring are:
Edge Refinement
Often RGBD footage will produce ratty images between things near and far in the scene. We're exploring using advanced image processing techniques, in combination with the metadata provided by the Portrait mode format, to help us clue getting rid of those ratty edges.
Gap Infill
Smartphone cameras can only take a photo from one position, meaning foreground subjects visually obscure the background behind them. This leaves cutouts in the background when projecting this data into the third dimension. We're actively exploring various gap infill solutions (machine learning, Poisson filters) to fill in those gaps.
***
Smartphone Portrait mode photos combine conventional photo data with 3D data, creating new opportunity for immersive experiences, but also requiring new software solutions for hologram editing. We're really excited about unlocking a whole new world of holographic photo editing and how exciting that can become, and with the ubiquity of smartphones that support Portrait mode photos, we're eager to see what people will start creating.
