Substance Painter VR: an Oculus Rift Prototype

  • R&D

Allegorithmic’s social media has been abuzz these last few days. Since we showed our Oculus Rift prototype at OC5 (the Oculus Connect conference), we’ve been receiving heaps of curiosity, excitement, and questions from all sides of the 3D community.

Allegorithmic’s Labs team is charged with pushing our boundaries by testing new concepts and developing prototypes. Sometimes that involves partnering up with other like-minded groups. Labs officially debuted at SIGGRAPH this year, demonstrating a live-streamed API prototype to feed RenderMan textures in real time, as well as some AI features that are shipping with Substance Alchemist, and our first steps with MaterialX.

For some time now, we’ve been intrigued by the potential of VR in an artist workflow. Our earliest prototype, spearheaded by Jean-Francois El Hajjar, first appeared on the scene over three years ago, when we revealed it at GDC 2015 in San Francisco. It was built for the Oculus Rift DK2 0.4.4; at that time, neither the progress of our work nor the Oculus SDK were quite ready for more general implementation which, along with the lack of Touch controllers, caused the project to be temporarily shelved, until the circumstances surrounding it became more favorable.

GDC 2015, featuring the first Painter prototype in VR

As part of an effort to make the Rift platform more appealing and approachable for DCC developers, this summer Oculus reached out to us for a reboot of this project. The partnership came at a good time for us; with a shiny new development kit in hand and new controllers, we were able to revamp our old prototype. Thanks to recent advances in both Substance Painter and the Oculus platform, Labs succeeded in developing a way to see and paint models in VR.

Substance Painter is a very powerful texturing tool. As such, it provides far too many options to port into VR without a deep UX study and a lot of UI work, so we started to rely extensively on Oculus Dash. But it was impossible to paint and use Dash at the same time – relying on it caused too much context switch, and created inefficiency within an artist workflow. Bringing no small expertise in painting, we collaborated very closely with our Oculus friends on their newer effort, Hybrid Apps.

This new software development kit dramatically lowered the entry barrier for using Substance Painter in VR, because it allowed us to embed one or more floating panels into the VR experience, mirroring the more traditional Qt-based panels. Having those available and freely arrangeable, we were simply able to map the most important set of features on the Oculus Touch controllers, such as selection of texture sets, viewport tumbling, painting, brush pressure, size, hardness, and tilt – plus, of course, undo and redo. With just a few weeks, we had a nearly functional Substance Painter in VR.

Robin Faury, testing one of his prototypes. You don’t actually have to lie down to paint with Substance Painter in VR.

At this point we were already aiming to show something for Oculus Connect 5, to showcase the new Hybrid App capabilities. With some time crunch, just to make things more fun, our own Jérémie Noguer was able to pull together a good paint pass on the classic robot from the Oculus First Contact demo. Armed with that asset, named Toby, and an experimental ‘Labs’ build, we recorded some footage in classic Substance Painter as well as in VR. We also filmed the hardware transition from a pen to a Rift, and sent all of it to the Oculus team, just hours before the deadline. Then, OC5 happened, and our snippet of video was streamed with the main Keynote event! It reached countless artists, professionals, and VR enthusiasts (1.5 million views so far on Facebook alone).

Of course, the overwhelming reaction of viewers across the VR world is testimony to the power of Substance Painter itself, and a credit to its talented team. This response highlighted the fact that there is a demand for a quality tool for asset creation directly in VR.

It isn’t really possible to convey with a video the impact of VR painting on an artist workflow. Despite the availability of our familiar tools on floating panels, the overall experience is completely different. No matter the resolution of your traditional display, the realism you get out of the very same shaders and textures is immediately enhanced with an HMD, because of the stereoscopic reflections, the sense of scale that comes out of a full immersion, and the high-fps response of the parallax in the image when your head moves around. With VR, you really are transported into the same world as your creation, without the subtle barrier imposed by a mouse, keyboard, and GUI. You can truly paint your creation – you can see it, and really feel its presence.

The removal of boundaries between artists and their artwork – this is what Substance is all about, and what motivates here at Allegorithmic. And so we’re very excited about the future of this prototype.

Stay tuned for updates.

Read more