28th April 2016

Mixed Reality Video

One of the really fun projects I got to work on over the past few months was creating a mixed reality video to show off SteamVR and the HTC Vive. With virtual reality we had this big problem that it can be really difficult to convey what its really like to be in it. So far mostly the best we have done to deal with this is just spend lots of time doing demos. Now that products are out in public the number of people who can experience it in person have grown a ton but its still only a limited set of people that will actually get to try out the hardware for 30 minutes.

Meanwhile there are a ton of videos that capture the first person view that the player sees inside VR or else footage of the person standing in a room moving around (or maybe a split-screen with both at the same time). These can be a good start but the first person view swings around all over as the person looks around and doesn’t feel especially real.

A mixed reality video on the other hand can provide a much better representation of what its like to be a real person “inside” one of these environments. An external camera shoots the person playing in a green-screen environment and they are composed into the views generated by the game (but the game generates a view from the perspective of the camera, not the person’s head). Camera movement helps a ton too because it helps you feel the real dimensionality of the space the person is in as opposed to the person just moving around on a 2d background.

Check it out here-

One of the best things about this was working with the awesome Valve film team and getting a chance to experience a professional video shoot and the great ideas they had about what would make a compelling video. The goal from the beginning was to make something that felt really real and didn’t come across as the usual marketing gimmick video. So a ton of the planning around the shoot involved bringing in normal people and really just trying to capture their actual reactions experiencing VR for the first time and playing the games. The funny thing is that many of the best reactions didn’t make the final video because they were so over the top that they looked fake. They were actually totally real but its one of those things where people would never have believed it just watching a 5 minute video. Oh well.

posted in Mixed Reality, VR | 0 Comments

4th April 2016

HTC Vive Launches Tomorrow

Once again one of those “I haven’t posted on my blog in a long time” posts. I guess I’ve been busy. Tomorrow is a really big day- the HTC Vive launches! SteamVR (our bit of it) has already been in use by thousands of people so in some aspects its a bit less of a big moment (and some consumer deliveries already happened today). But tomorrow the press embargo lifts and all the titles launch (+ a little extra thing) so its a really big deal.

Those titles are a key part of it. One of the most exciting bits for me over the past year+ has been working with tons of developers on their stuff and getting a chance to see a bunch of these games (and other) early.

One of the interesting phenomina in the VR world is there is lots of “what is VR?” discussion. The classic one is panoramic 360 photos/video. I don’t feel a need to include/exclude things, but there has been an interesting discussion about whether these really count (since they don’t really work right as you move your head, etc).

But beyond that if you look at the bigger VR landscape there are a bunch of titles that are things that could basically be on a flat 2d monitor but just displayed in 3d. Those are fun and the 3d makes them feel fresh.

But some of the new stuff coming out tomorrow is really a different level. Room-scale and even more importantly motion controllers really transform this stuff. There are titles that are brand new things that aren’t like anything you have ever seen before. But then there is also stuff like Space Pirate Trainer. The thing about that one is that in some ways the game is like Galaga. A 35 year old game where there are space ships flying around above you in patterns and you have to shoot them. But Space Pirate Trainer transforms this into an entirely new thing- you point with your hands, you dodge, you have to look out for things coming from both sides, you get to block stuff with a sheild. Its not slightly fresh, its totally fresh and I think is a perfect way to reveal how much computing will be transformed by VR (and AR and in-between).

posted in Gaming, Technology, VR | 0 Comments

18th September 2015

VR Impact on Development

Another interesting aspect of VR is how it impacts the development process and how you structure your workplace when building VR experiences. Valve’s normal “cabal” structure is pretty well documented elsewhere but we have had to adapt how we organize our desks for VR. It certainly helps that all of our desks are on wheels so we could just move things around to adapt.

The key thing for us is that its important for every developer to have a quick ability to stand up from their desk and try out the code they are writing. I should say that depending on the thing you are working on probably 50% of the time or more we just end up putting on a headset at our desks and trying things out there. But when working on things that are more about interactions its important to be able to stand up and move around a bit. We have structured areas so that we have a bunch (4-8) desks that surround a common area that can be used for this testing. So all the desks are around the outside and when you need to try something out you just stand up and use the space.

Depending on the group these spaces are different sizes- some of the people working on things that need more moving around have spaces that more closely approximate the maximum tracked area, but I rarely need that and am sitting with a few others in a smaller space. Each area has its own pair of base stations that we can all share but we each have our own HMD which I realize is a luxury that most developers don’t have yet (balanced by the fact that we have to try out new builds of stuff all the time to make sure it works before it gets to others). Right now because base stations can interfere with each other each area is curtained off from other areas. With the final product this shouldn’t be necessary since the sensors will be able to reject incorrect signals.

Of course with shared spaces its important to share- we can’t all be testing and bumping into each other at the same time. Also there is an important rule that the person with the HMD on their face has right of way. If you are moving through the area its your responsibility to avoid them since they can’t see you (for now).

posted in Developers, Technology, VR | 0 Comments

25th August 2015

VR Performance and CPUs- Cores vs Frequency

I feel like I’ll probably write a bunch about performance and VR as we go along since its such a complex topic and there is so much stuff we still need to learn. PC and gaming performance has always been an interesting topic but whereas in the past it was mostly about “can I average 50fps or 55fps on this game?” now it becomes “can I consistently get 90fps 99.9% of the time so I don’t get sick”. 100fps doesn’t matter, 92fps doesn’t matter, but 89fps just sucks. 90fps becomes a really sharp performance line.

So far we only have really preliminary performance data since it depends so much on the content and all of that is in-progress and hasn’t really been optimized. People also haven’t done the work to have their experiences scale up and down depending on the PC they are running on. For example with GPU performance you can do a lot by changing the size of the render target which is normally 1.4x the size of the real physical panels in each dimension (so a total of 1.96 times the pixesl). Having at least 1.4x scale gives you ideal anti-aliasing and sharpness when the panel distortion is applied but if you don’t have enough GPU performance you could reduce that at the expense of some sharpness and graphical quality. Not ideal but it sure beats missing frames.

If CPU becomes the limiting factor your choices are a little more complicated. For one games are traditionally poorly multi-threaded and thus don’t take good advantage of additional cores which is the main CPU performance improvement lately. When dual-core CPUs first came out most games didn’t take advantage of them at all and while the situation has improved quite a bit recently few games seem to take full advantage of 4-core CPUs never-mind some of the 6-core machines that work great as development boxes (compilers LOVE 6-cores now!)

One of the issues in picking the ideal CPU is that CPUs are typically limited by their power consumption / heat generated. So a 6-core CPU is usually limited to much lower frequencies than a 4-core CPU. While it can have overall much more computing resources if the given task isn’t able to keep at least 6 threads busy it will become gated on the performance of a single core which is higher in the chip with fewer cores. So I suspect for many gaming scenarios the high end 4-core CPUs will be better for now than the otherwise better 6-core ones until games can make better use of threads. I’m looking forward to trying out the new Intel Core i7-6700k which is the new 4-core 4.0ghz Skylake CPU that should hopefully be the fastest yet for these scenarios.

posted in Graphics, Performance, Technology, VR | 0 Comments

18th August 2015

VR API support in Games

Various recent discussions have been somewhat confusing about what it takes to add VR support to a game, and specifically what it takes to support multiple VR devices. As usual there are several layers to the issue so some degree of confusion is pretty understandable.

At its most simple level VR support is provided by two key aspects-
1) rendering a stereo view of the scene into two textures (or one texture with one eye on one side and the other on the other side) and handing that off to the VR api. At that point the VR API does a bunch of complex stuff including some corrections, distortion for its optics (which can be calibrated for individual devices) and composting additional system UI. But none of that latter stuff is something the app developers have to worry about.
2) Updating the head position as the HMD is tracked. The underlying VR API should provide the game with a “pose” that represents the exact location and orientation of the head so that as you move your head around in the real world your view will exactly correspond.

At the above level it should be very easy for VR game authors to target multiple vendors headsets. If you are using Unity or Unreal Engine, support for both the Oculus and OpenVR SDKs are built-in / available with the addition of a simple plug-in. Better yet OpenVR is designed with a driver model for different hardware and our goal is to support any major HMD if you use it (although we are limited in our ability to promise specific support because vendors can change their APIs in breaking ways at any time and then we need to update).

Beyond the basics of the headset and scene rendering are other issues around the interactions that the game supports. If your game doesn’t use controllers and is a basic seated or simple standing experience, it should pretty easily work with any headset. In a similar fashion if you use non-tracked controllers like the Xbox or Steam controllers, anyone can buy one of those and use it with any headset. Tracked controllers become a more complicated problem- they each have some more specific behaviors and are tied to the specific tracking system. So in the short run, it would be hard to mix and match the controllers and if a specific game expects the HTC Vive trackpad input as part of its interaction, or another game is designed around the Oculus Touch specific features those might be harder to support across both platforms.

Our goal with SteamVR/OpenVR all along has been to develop technology and make it available to others in the VR community so that we can have high quality VR that doesn’t make people sick and where our customers can buy any VR equipment and enjoy their games on it. We hope Lighthouse will become a standard for tracking so that you can use controllers from many vendors (in many styles) to interact with all your games, but the OpenVR APIs can also work with non-lighthouse tracked controllers. Hopefully the VR industry can avoid following the early days of the 3d-accelerator market where games were created that only worked with one vendors hardware.

posted in Technology, VR | 0 Comments

11th August 2015

Cloudhead on Locomotion in VR

Cloudhead Games has some great ideas about locomotion in VR in this video.

This video brings up a couple of interesting topics worth discussing a bit more.
The first is about some details of the chaperone system. The Cloudhead guys discuss how they are showing some hints in game about the bounds of your real world space and I thought it would be interesting to discuss that in the context of the overall system.

Chaperone exists first and foremost to help people feel comfortable about moving around in room scale VR without having to worry about hurting yourself by running into something in the real world. One interesting aspect of this is that the chaperone system can really help the feeling of presence because once you get used to it you feel much more free to move around in your space. We do a ton of play testing and demos and pretty commonly see the first couple of minutes people move around with a lot of hesitation but once they get comfortable they let the experience they are in take over.

Within the chaperone system we think of two boundaries. The hard-bounds are the real limits of the physical space. If you are in an empty room, that would be the walls. If the room isn’t empty it might be some artificial line that represents the furniture. But the point is hard-bounds are real “do not pass” line. Hard-bounds are implemented in the OpenVR system and when the player gets near them we will draw the shape of the room over anything the current experience is drawing right in the compositor. This is actually somewhat intentionally presence breaking. If you are about to collide with something we want to bump you out of the immersive experience right away.

We also tried experimenting with using just a partial wall in front of you for the hard-bounds to tone it down a bit. I didn’t like that approach much because once I was getting into those limits I really wanted to see the shape of the whole room to help reorient myself.

Soft-bounds however are another matter. Soft-bounds are another set of lines inset a bit from the hard-bounds and they really represent the space you should be inside most of the time when playing. When you are standing inside the soft-bounds, you shouldn’t be able to easily wack things outside the hard-bounds. Soft-bounds aren’t drawn by the system because we don’t have a good way to do it automatically and keep you immersed in the experience you are in. However each experience can ideally represent the soft-bounds in some way that is consistent with its own look and that doesn’t really disrupt your overall presence. One great example of this is TheBlu from WEVR. You are standing underwater on the deck of a ship and its safe to walk anywhere up to the railing in one direction and up to a bunch of junk and wires in the other direction. If your play-space is smaller they could pretty easily move the junk and wires so that the clear area of the deck still represents the safe space. Other games might need to use things like subtle glowing lines on the floor (as seen in the Cloudhead video).

So the key is- use subtle cues to give the player hints about the right place to play. And when they move outside those limits give them less subtle warnings to keep them safe and help reorient themselves in the space.

The Cloudhead video also shows some great ideas about dealing with virtual walls. Obviously in VR you can’t prevent the user from moving the camera into some object just because its supposed to be solid in the virtual world. Remember that the camera IS their head, so if they move their head you have to move the camera. But it doesn’t mean you can’t do something about it. Now, sometimes its pretty fun to just let people put their heads inside your geometry and see the cool insides (people do this all the time with the robot in our Aperture demo and its a lot of fun). But if you don’t want to allow that you can either blur or fade the world when they put their head into the object. I especially like the Cloudhead solution of teleporting you back to some safe place if you stay in there for a while- the problem with just fading or bluring is it can become disorienting to not have that visual reference so it can be uncomfortable or actually difficult to find the correct direction to back out.

Finally I have head some reactions to the video of people expecting that the teleportation for locomotion will diminish presence. I’m sure this varies a bunch from person to person but for me at least it ends up very natural. Its hard to describe but after using it for hours I end up actually teleporting around fairly quickly and my brain doesn’t even think about it that much anymore. In contrast when I try a VR title that moves me around by gliding it feels very unnatural- something about the smooth motion seems wrong (and then I get sick and things feel even more wrong.. )

It all goes back to part of what is so exciting about working on VR right now- no one really knows the right way to do all this stuff. The space is wide open for people to come up with new ideas and test them to see what really works.

posted in Technology, VR | 0 Comments

27th July 2015

Quick thoughts on GPUs for VR

I see lots of speculation around about what graphics card you need for good VR experiences. At this point we are at least a few months away from availability of commercial VR systems for PCs. As such the advice I can give is pretty easy right now- wait if you can! If you absolutely positively need to buy a new graphics card now, go ahead. But if you can wait until you are actually receiving your VR headset things will be cheaper and there will be more information about upcoming cards, relative performance for VR scenarios and the needs of various titles needs.

In the end the answer won’t be straight-forward because it will all depend on content. Can you run VR with a HTC Vive with an older graphics card? Probably if all you want to do are more simple experiences. Heck, you could almost get the required 90FPS on an integrated GPU if your scene is just a simple cube (but maybe not quite). But the key thing is that you REALLY NEED to make 90fps. All the time. Techniques like async time warp are interesting but they have really visible artifacts when used with positional tracking and if you have tracked controllers they will be VERY noticeable (since every timewarp your controllers will glitch in their positions). Good experiences in VR are more about consistent smooth motion that 100% matches what you expect even more than realistic rendering. For example one of the demos we show is Job Simulator where everything is a cartoon representation of objects. It doesn’t look like reality but the Owlchemy guys did such a good job with it that it FEELS like reality.

So what am I going to do for my personal rig? First of all, wait as long as possible. Second, I’m not going to cut corners. Like most tech VR equipment will great much less expensive in the future. But this is the first generation of real products and its not going to be inexpensive for a quality setup. In the past I have never bought a GPU for much over $300, but I’m expecting to spend in the $650-range for either a 980ti/equivalent or maybe dual slightly smaller GPUs since stereo rendering can do such a great job on dual-GPU setups. I know its a lot, but then again 15 years ago I used to have to spend $4000 for a high-end PC and the whole system with an amazing GPU should be less than $2000 now. Until now the difference between that $300 GPU and something better was really hard to tell for most people, but with these VR experiences it becomes worth it.

posted in Graphics, Technology, Virtualization | 3 Comments

27th July 2015

VR Growing Pains

One of the things we are very passionate about at Valve is helping the industry create a whole new set of VR experiences that people love. We want to make sure that the exposure that the world gets to VR is really great and it doesn’t make people sick/sore/unhappy.

Surprisingly this is actually somewhat controversial. While many people can get sick easily if they are in VR experiences that move the world around them without the player actually moving, there are some people who are (mostly) immune to it. They can play some of the FPS games adapted for VR or just “tough it out” to try experiences. I’m the last person who is going to tell them they can’t do that to themselves, but I do get concerned when these people are excited about sharing those experiences with their friends and I certainly advise software developers working on new VR titles to make experiences that will be enjoyed by a wide audience, not just that 20%.

Another way to think about it is you don’t want the reviews of your game to read like these (actual reviews)-
“New X Title ‘Y’ Is Beautiful and Engaging and It Makes My Neck Hurt”
“With a little luck, (and dramamine for motion sickness) the VR mode will make it to the final release”
“I intentionally limited the amount of barrel rolls and hard turns as to avoid the dreaded ‘hot sweats’”
“In the demo I played stick-yaw was the method of turning, which made me a little motion sick by the demo’s end”
“X will test your skills… and potentially your stomach”

The bottom line is that the real audience isn’t going to tough it out, train themselves to avoid sim sickness or anything like that. If your game doesn’t get this right your audience will be extremely limited. There are already some fairly well known techniques to do this right but we are all learning more- again, the key is honest testing.

posted in Technology, VR | 0 Comments

7th June 2015

Macs and VR

The funny thing is that Macs should be great for VR. On average they tend to have slightly better graphics than the typical PC because Apple controls the ecosystem and has always needed good baseline graphics performance to support the visuals in their OS.

HOWEVER, quality VR is really hard to run on middle of the road GPUs. And to be clear, middle of the road in this case is not 50% of the performance of the high end (what you would see from a 970 or 980M). Its 10%. 70% of Mac gamers according to the Steam Hardware Survey are running on laptops. And really no Macs at least as configured by Apple short of the $3999 MacPro models have enough GPU to make 90 frames per second with an HMD for any reasonably complex content. The new Retina 5k display iMac can be gotten with an MD Radeon M295X that should be good enough but that is over $2500 and doesn’t even have an HDMI output so hooking up an HMD would need an adapter that might cause some interesting issues.

By comparison its pretty easy to build out a PC for about $1500 that has a top of the line Core i7 and a GeForce GTX 980 that should be equivalent to what we are using for demos. I should say that most of the PC OEMs don’t make this easy at the moment- good luck for example finding a system on the Dell site with reasonable specs without getting a very expensive Alienware. Some of the OEMs that let you customize a bunch more are a more reasonable bet for now. Hopefully as VR becomes more of a think on the marketplace both Apple and the PC OEMs will catch on and offer some great configurations optimized to be good for these scenarios (which also means they will be good general gaming PCs).

posted in Apple, Technology, VR | 0 Comments

6th June 2015

VR- Getting Sick

The topic of getting sick in VR (often called “simulator sickness”) is a complex one. The simplified core of the issue is that when your visual perception doesn’t match what your body (and especially inner ear) feels, many people feel sick. This is obviously not exclusive to VR- it can manifest in cars, airplanes, spaceflight, amusement park rides and more. Note that whatever the mechanism is for this in the brain isn’t 100% tuned to all exact physical experiences. Many people get sick on a roller coaster even though their motion perfectly matches what they see- just the brain didn’t evolve to be prepared for tight accelerating turns, changes in the gravity and so in those cases the mismatches between reality and brain-reality are enough to cause the problem.

Some people even get sick playing first-person video games without VR. The types of motions in a typical FPS with lots of dashing around, quick turns, and strafing movements can be especially difficult for people who experiences these issues.

In VR I think of two main sources of mismatch between what your body feels and the photons entering your retina. Both can be equally important for VR experience designers to consider and both can be very complicated to get right. System-mismatch is the mismatch caused by the core VR technology. At some basic level a VR system is measuring your head’s current position, velocity and acceleration and rendering a frame. Ideally that frame will be positioned so that it exactly corresponds to your head position at the moment in time the photons hit your retina. The other type is content-mismatch. This is by-design movement of the camera/head controlled by the experience that doesn’t exactly match the player’s physical movement.

For the fundamental technology we are working on at Valve we spend a lot of effort on system-mismatch. There are a number of techniques to address this, but they involve accurate tracking of head position, prediction of future head position, reducing latency in the rendering pipeline, deferring locking in head position in the rendering pipeline, adjusting rendering at the last minute, making sure rendering actually makes framerate (since there is no worse added latency than missing a frame totally) and finally reducing latency between your GPU and the actual photons being emitted (HDMI has a fair amount of latency and this relates to why wireless displays are very difficult).

As I said, most of these are issues for the core VR system. As a content/experience/game developer there isn’t that much you can do to change the tracking accuracy of whatever VR hardware you are using. Your main challenge is to never miss framerate. On the HTC Vive we run at 90hz so overall there are 11ms per frame. But there is plenty of overhead between various system things, the compositor, GPU scheduling, etc, so realistically you only have about 8ms to render your frame. It is critically important to design your content so that it fits within budget 100% (or at least 99.9%) of the time. Ideally game engines will help since its really hard to tune content to be consistent- most environments have something simple most of the time punctuated by some really complicated stuff occasionally (when all the explosions happen or something). If your game engine can automatically adjust and reduce render target sizes, AA, texture mips, etc, that can help a ton.

Content-mismatch is however a more complicated issue. In general I’m a fan of never making your players sick. Its true that there are some people who are less susceptible but its not clear to me that they are actually comfortable after spending a bunch of time in an experience that is questionable. I’d love to do some research where we wire some of them up to HR, breathing, and other monitors and have them play an experience that pushes the boundaries for a full hour. Personally I’m middle of the road- I’m far from immune to getting sick but not nearly as bad as some. Part of what confuses some of the debate so far is that so few people have extended time in VR. When I’m playing an experience that moves the camera I’m usually fine for 2 or 3 minutes (typical demo length), feeling just a little “off” at 5 minutes and it doesn’t get bad for a bit longer. But VR is still not going to be successful if 80% of people are feeling even a little uncomfortable after 5 minutes. They might not even realize they feel sick, but they certainly won’t enjoy the experience the same way they would if they were in well designed experiences that didn’t have these issues.

With all of the above preface, its useful to discuss what you can actually do. Most of the demos that Valve showed at GDC and since then had almost no forced-camera movement. We wanted to be especially careful to make sure people had great experiences while we are still learning what works. At this point I don’t think we can definitively say that we know what works, but there are a bunch of techniques that look promising.

In general it seems fine to have the player in a slow-moving elevator with good visual references. CloudHead Games does this well in The Gallery demo. Cockpits seem to help so driving/flying airplanes/etc look like they might work when done carefully (with the caveat that many people get sick in the real thing with these scenarios). The agency of directing the vehicle (car/airplane) yourself seems to help some. I have also seen good results with locomotion with teleporting- I often build prototypes that let you use one of the controllers to point to where you want to be and press the button to teleport yourself there to move over distances greater than the physical space you have available.

RoadToVR has some interesting stuff about experimenting with locomotion at the Austin VR Jam here.

posted in Technology, VR | 0 Comments