Key Concepts

Before we can understand the issues related to 360°, 3D, AR and VR technologies, we have to understand some key concepts.

Immersion and Presence

The goal of 360°, 3D, AR and VR technologies is to immerse users in an environment, so that they feel they have been “teleported” to this new locale and are actually present in this new world. Achieving immersion and presence requires that the brain be fooled by the senses into believing it is somewhere that it really is not.

Here are descriptions of immersion and presence by some other sources:

Reality Technologies:

Total immersion means that the sensory experience feels so real, that we forget it is a virtual-artificial environment and begin to interact with it as we would naturally in the real world.

…

Virtual reality immersion is the perception of being physically present in a non-physical world. It encompasses the sense of presence, which is the point where the human brain believes that is somewhere it is really not, and is accomplished through purely mental and/or physical means. The state of total immersion exists when enough senses are activated to create the perception of being present in a non-physical world.

…

a sense of immersion (i.e. convincing the human brain to accept an artificial environment as real).

iQ by Intel:

… presence: “The unmistakable feeling that you’ve been teleported somewhere new.

VR Lens Lab

… presence. That is, the ability to take you somewhere other than where you really are, and trick your mind into believing it.

Jonathan Strickland at How Stuff Works

In a virtual reality environment, a user experiences immersion, or the feeling of being inside and a part of that world.

Field of View

VR Lens Lab provides the following description of field of view (FOV) (see Figure 1).

Field of view, or the extent of the observable environment at any given time, is one of the more important aspects of virtual reality. The wider the field of view, the more present the user is likely to feel in the experience.

Monocular FOV describes the field of view for one of our eyes…

Binocular FOV is the combination of the two monocular fields of view in most humans. When combined they provide humans with a viewable area of 200°-220°. Where the two monocular fields of view overlap there is the stereoscopic binocular field of view, about 114°, where we are able to perceive things in 3D.

While a wider field of view is important for immersion and presence this stereoscopic binocular field of view is where most of the action happens every day and also in virtual reality headsets.

Figure 1: Horizontal Field of View

 

Source: VR Lens Lab

In addition to the horizontal aspect of field of view, field of view also has a vertical aspect, that is, how far up and down we can see (see Figure 2).

Figure 2: Vertical Field of View

 

Source: Stack Overflow

 

360°

Anurag Syal provides the following description of 360° technology in “Panorama vs 360° vs 3D vs VR — The Big Difference”:

A 360° shot is simply a panorama that covers the full horizontal field of view around a person.

... When played back in a VR headset, it can feel immersive — but what you see is real footage, not a simulation. You can look around and even feel like you're exploring the scenery, but you can't interact with it much and you can't travel within it.

360 videos are presented in 2D. This means that the image has height and width, but will always lack depth. Think of it like taking a big poster and wrapping it around you. You can turn around and look at the different pieces of the image, but it’s still just a flat picture.

Figure 3: Panorama Photograph

 

Source: Anurag Syal, “Panorama vs 360° vs 3D vs VR — The Big Difference”

 

3D

The Free Dictionary describes 3D visualization as follows.

The creation of 3D prints, images and movies is accomplished by capturing the scene at two different angles corresponding to the distance between a person's left and right eyes (roughly 64mm). When the left image is directed to the left eye and the right image to the right eye, the brain perceives the illusion of greater depth... A corresponding pair of 3D eyeglasses directs the images to the appropriate eye.

An example of a 3D camera (a Lenovo Mirage) is displayed in Figure 4.

Figure 4: 3D Camera

 

Source: Lenovo

Examples of images captured by 3D cameras are displayed in Figures 5 and 6.

Figure 5: Images Captured by 3D Camera

 

Source: IMAX SoS Planet

Figure 6: Images Captured by 3D Camera

 

Source: LG

 

Augmented Reality (AR)

Augmented reality involves projecting digital content onto the real world. As Augment describes it,

Augmented reality (AR) is a technology that layers computer-generated enhancements atop an existing reality in order to make it more meaningful through the ability to interact with it. AR is developed into apps and used on mobile devices to blend digital components into the real world in such a way that they enhance one another, but can also be told apart easily.

Examples of AR systems include Microsoft's HoloLens system and Magic Leap’s Magic Leap One.

AR is interactive to the extent that the digital projects change either with users’ actions or with the exterior environment.

 

Virtual Reality (VR)

Virtual reality involves transporting users into a wholly virtual environment. As Augment describes it,

Virtual reality (VR) is an artificial, computer-generated simulation or recreation of a real life environment or situation. It immerses the user by making them feel like they are experiencing the simulated reality firsthand, primarily by stimulating their vision and hearing.

As Reality Technologies describes it,

Virtual Reality, noun: A realistic three-dimensional image or artificial environment that is created with a mixture of interactive hardware and software, and presented to the user in such a way that the any doubts are suspended and it is accepted as a real environment in which it is interacted with in a seemingly real or physical way.

iQ by Intel indicates that successful VR experiences, that is, those that achieve “comfortable, sustained presence” require “a combination of the proper VR hardware, the right content and an appropriate system.”

Examples of VR systems include HTC Vive, Oculus Rift, and PlayStation VR.

 

Interactivity

The feature of AR and VR that makes them so much more desirable than simple 360° and 3D media is user interactivity with the media. As Anurag Syal describes it,

Various forms of visual media would never achieve what VR can, because they lack the most desirable of all elements required for entertainment — Interactivity.

In other words AR and VR users are active participants in their activities, not merely passive spectators. Joshua Goldman and John Falcone make this point in “Virtual reality doesn't mean what you think it means”:

Through the use of VR headsets loaded with sensors that track your head and eye movements, you're able to interact with and navigate through different environments as if you were actually in them. This is the main thing that separates VR from immersive multimedia content where you're more of a spectator than a participant.

And, as Jonathan Strickland iterates, it is user participation that enables users to become fully immersed and achieve presence in their environment:

For immersion to be effective, a user must be able to explore what appears to be a life-sized virtual environment and be able to change perspectives seamlessly. If the virtual environment consists of a single pedestal in the middle of a room, a user should be able to view the pedestal from any angle and the point of view should shift according to where the user is looking.

…

True interactivity also includes being able to modify the environment. A good virtual environment will respond to the user's actions in a way that makes sense, even if it only makes sense within the realm of the virtual environment.

Unfortunately, enabling user interactivity does not come without a steep cost. Ben Grossmann, co-founder and CEO of Magnopus, provides more detail on the cost of interactivity in his interview with Foundry Trends:

Of course, giving them [users] more agency makes it exponentially more expensive. Creating a 30-minute linear story will be one cost but a story in which the audience can decide on, say, five different outcomes, means producing 250 minutes.

Figure 7 categorizes 360°, 3D, AR and VR technologies by whether they involve the real world or computer simulations (this line seems to be blurring) and whether technology users are passive/spectators or interactive/participants.

Figure 7: Categorization of Technologies

 

 

VR Industry Games

The VR technology industry is actually involved in a couple of different games. One is a free-riding game by developers of 360° and 3D technologies, who are passing their technologies off as being true VR, to satisfy user demand for true VR content. The actions of these developers are causing users who don’t understand the difference between the “lesser” technologies and “true” VR to discount the value of true VR. This will make it more difficult for developers of true VR to convince users to adopt true VR.

Another game is a classic hardware-software game: Users have been reluctant to adopt VR hardware because there’s not enough content available. Developers of content for VR systems, on their part, have been reluctant to provide content because it’s expensive and difficult to do so.

Free-Riding

The Problem

Non-VR technologies are free-riding off of virtual reality, which is diluting the value to developers and consumers of true virtual reality technologies.

VR contains many different features, including 360° field of view, 3D imagery, and user-interactivity. Developers of “simple” 360° technologies and 3D technologies often refer to their technologies as being VR. This attracts (uninformed) consumers who are lured by the sex appeal of “virtual reality.” In other words, developers of “simple” 360° and 3D technologies who refer to their technologies as VR are free-riding off the value of true VR.

Dealerscope makes the point that one of the reasons 360° and 3D technologies are able to get away with passing their technologies off as VR is that users are hungry for VR content. Unfortunately, though, there’s simply not enough true VR content out there to satisfy demand:

A lot of the reason for this rush to bring 360 video to VR platforms isn’t because it’s a fit, but because despite the near constant buzz about VR, this is still a nascent platform that is desperate for truly immersive video experiences.

Consumers who aren’t familiar with true VR, or with the limitations of simple 360° or 3D technologies relative to VR, come away believing they have experienced true VR. Subsequently, those consumers then discount the value of true VR, believing it only offers the benefits of simple 360° or 3D technologies.

As Will Smith states in “Stop Calling Google Cardboard's 360-Degree Videos 'VR'”:

360 video is as far from real VR as seeing the Grand Canyon through a Viewmaster is from standing at the edge of the canyon’s South Rim.

With technology as potentially polarizing as VR, I worry that the slightest hiccup will have a negative impact on people’s perception—and adoption—of that tech.

Foundry Trends iterates the point that developers who are trying to pass off 360° and 3D technologies as VR “just aren’t really helping” users appreciate the value of true VR.

Essentially, you end up delivering a limited experience to mobile users, who end up thinking this whole VR thing is a bust… a lot of mobile experiences just aren’t really helping the cause.

The situation is especially problematic for VR developers because VR is an experience good: it’s not enough to describe the benefits and features of VR. You can’t convincingly advertise the benefits of 3D using 2D media. Rather, consumers have to actually experience it to understand its true value. As Hunter Sappington puts it in “The VR Experience: Challenges for a Growing Market,” “Because the value of VR is best understood when experienced, headset demos are key to VR adoption.” The problem is that consumers of simple 360° and 3D technologies who believe they have experienced true VR will be reluctant to make the effort to experience true VR, because they have already discounted its value.

Complications

The problem of discounting the value of true VR technologies based on experiences with 360° and 3D technologies is that true VR technology hasn’t yet been fully developed. It’s difficult for developers of true VR to take action to reclaim value for true VR until well-developed VR is actually available. Of course, not all consumer reluctance to adopt VR technologies is due to users of 360° and 3D technologies discounting the value of VR. Rather, much of the consumer reluctance to adopt is due to the less-than-ideal state of currently available VR technologies.

Evidence that current VR systems aren’t fully-developed:

Alan Buzdar on “Why The Oculus Rift Still Isn’t Worth It”:

Unfortunately, after hours in virtual reality the initial high from the flashy visuals and new interface wears off and you’re left with the stark truth: there is no substance. As it stands today, virtual reality is a brilliant new interface with no compelling reason to use it. It’s as if developers were so focused on making their games beautiful and thinking of novel ways to use the interface, they forgot that games need to be fun. Even games that are supposed to have stories are very shallow and often times can be played through in 15 hours or less.

…

Motion Sickness. This problem is self-explanatory so I won’t go into too much. But, I doubt there exists a human who could play first person Minecraft on their Oculus without throwing up within 10 minutes.

From Natasha Lomas, “This VR cycle is dead”:

One veteran games industry developer who works for a top tier games publisher — including advising on the platforms the company should support — and who previously took an early interest in Oculus and VR in general; excited to experiment and bullish on its prospects back then, now tells me flatly when I ask about the current status of VR: “Come back in five years.”

Here’s what else he has to say:

It feels like the second-wave bubble of VR is about to burst.

The technology is incredibly exciting, and it has the potential to be huge one day. Unfortunately neither the hardware nor the software is ready for that yet.

In the meantime, a potential consequence of consumers’ discounting the value of VR – for whatever reason – is that VR developers may have problems getting funding.

Potential Solutions

Developers of simple 360° or 3D technologies are free-riding off the appeal of true VR to attract users to their simpler technologies. In the process, they are diluting the value of true VR. In this case, the onus is on developers of true VR systems to take action to help consumers understand the difference. The situation is similar to that originally discussed by George Akerloff in his well-known analysis, “The Market for Lemons”.

Perhaps the least costly action VR developers can take is to advertise the differences between VR and simple 360° or 3D technologies. That is, they can actively explain to consumers how their technologies differ from (and are superior to) simple 360° or 3D technologies.

Perhaps the easiest way to make the problem go away, though not the easiest to actually accomplish, is to come out with a killer app that highlights the value and benefits of true VR.

 

Hardware-Software Game

The Problem

VR hardware requires software, or content, to generate value, but content providers have lagged in providing VR content. In turn, lack of content has delayed system adoption by users. That is, VR technology is experiencing what I call a classic “Hardware-Software Game” (see Figure 8). To induce adoption by consumers, hardware developers need content providers to provide content. However, content is expensive to produce. And given the fragmentation of the hardware market, together with the rapid technological progress, good content is difficult to find. In turn, lack of available content impedes system adoption by users.

Figure 8: Hardware-Software Game

 

Users’ Barriers to Adoption

The main barriers to adoption of VR systems by Users are (i) VR systems are too expensive, (ii) especially given the lack of a real value proposition, together with (iii) the fragmentation of the market.

In “Current Challenges of Virtual Reality, Explained,” Jon Buck remarks that “Hardware for capturing 3D images is rudimentary at best, unless users seek out the uppermost level of hardware at extremely high price points.”

Hunter Sappington noted that in a survey of U.S. broadband households, “more than half … reported that they did not think the experience would be worth the extra expense of buying a headset.” At the same time, Hunter Sappington noted that the market for hardware systems is fragmented, with several different platforms available in the market. So even if there were an overall desire for users to adopt, “there are no ecosystems sufficiently developed to drive headset sales on their own.”

 

Content Developers’ Barriers to Providing Content

There are several hurdles impeding developers from supplying content for VR systems: high quality images needed to create content aren’t widely available, markets are fragmented across incompatible technology platforms, the technology is evolving rapidly, and there aren’t enough people with the right skills to create content.

Expensive Inputs. High quality images are needed by content developers to create good content; however, such images aren’t widely available. Lack of available has hindered provision of content for VR systems. As Jon Buck explains,

The popularity of VR has been hindered, however, due to lack of access to high quality 3D images for VR developers. Image repositories often provide some images for free, but those images are of lesser quality or have already been over-used.

…

Developers are forced to either use low-quality images or develop the images themselves at a substantial cost, often pricing the bulk of the VR market out of their product.

Market Fragmentation. Platform fragmentation across technologies has also hindered the provision of content. It’s time-consuming and expensive to adapt content for different systems. As Middle VR explains,

If you run it on a different hardware … You then have to modify your application to take into account the new trackers, the new screens, the cluster synchronization. You will also probably need to modify the interactions, because you don’t necessarily have a perfectly equivalent hardware, or tracking volume.

And as Hunter Sappington describes the problem,

Technology Fragmentation: VR market players have been experimenting with a variety of tracking, input and content technologies, with no common standards followed by the entire industry. As a result, much of the content produced remains constrained to particular headset hardware, and conversion to alternative hardware often is costly. This fragmentation makes content development expensive and limits content distribution. With no dominant VR headset maker and little prospect of near-term consensus, the lack of standardization will remain an issue for the next few years.

Rapid Pace of Evolution. Rapid technological innovation is making it costly and difficult for content developers to keep pace. From Jon Buck,

Further, 3D technology is rapidly improving, often making images obsolete very quickly. Image creators must continuously invest funds into image creation hardware that is extremely costly in order to keep abreast of the marketplace.

Lack of Skills. Providing content for VR systems requires a different skill set from that required for previous technologies. Not enough people have yet acquired the skills needed to keep up with demand for content. Nick Ismail describes this problem in “What’s holding virtual reality back?”:

Virtual reality expertise is an increasingly sought-after skill sets in the job market and reports show that in the second quarter of 2017 the demand for online freelancers skilled in VR grew far faster than for people with any other expertise. The challenge the industry faces now is developing the talent, upskilling content producers and re-skilling those who are curious to learn about virtual reality programming.

 

Hardware Developers’ Barriers to VR Development

iQ by Intel notes that “Believable experiences in virtual reality come at a price. ‘The reality is that VR is more demanding than any other type of gaming we’ve ever had,’ said Palmer Luckey, Oculus founder.”

High quality VR technology creates a sense of immersion and presence for users. To achieve a sense of presence, developers must attain new levels of quality in several areas.

Level of Detail. iQ by Intel notes,

In virtual reality, players can lean over and look at something up close. Such scrutiny requires better graphics to be believable. Players have to be able to see the hairs on a dog.

User Interactivity. Jonathan Strickland notes,

True interactivity also includes being able to modify the environment. A good virtual environment will respond to the user's actions in a way that makes sense, even if it only makes sense within the realm of the virtual environment.

And Foundry Trends adds,

Of course, giving them more agency makes it exponentially more expensive. Creating a 30-minute linear story will be one cost but a story in which the audience can decide on, say, five different outcomes, means producing 250 minutes.

Frame Rate. Reality Technologies describes the concept of frame rate and how frame rate demands of VR exceed those of previous technologies.

Frame rate refers to the frequency (rate) at which the display screen shows consecutive images, which are also called frames. Television shows run at 30 frames per second (fps) and some game consoles run at 60 frames per second (fps). In virtual reality, a minimum frame rate of approximately 60 frames per second is needed to avoid content stuttering or cause of simulation sickness. The Oculus Rift runs at 90 fps, providing Oculus Rift users with a very lifelike experience. Future Frame rates for virtual reality headsets are set to inevitably continue getting faster, providing for a more realistic experience.

Latency. Reality Technologies describes the concept of latency and how latency demands of VR exceed those of previous technologies.

Latency refers to the amount of time it takes for an image displayed in a user’s headset to catch up to their changing head position. Latency can also be thought of as a delay, and is measured in milliseconds (ms). In order for an experience to feel real, latency usually needs to be in the range of 20 milliseconds (ms) or less. Low latency, or very little delay, is needed to make the human brain accept the virtual environment as real. The lower the latency, the better. The higher the latency, a noticeable and unnatural lag may set in, consequently causing simulation sickness for the user.

Tracking. Reality Technologies describes how user tracking demands of VR exceed those of previous technologies.

Tracking handles the vital task of understanding a user’s movements and then acting upon them accordingly to maintain full immersion in virtual reality.

… Head-tracking technology must be low latency in order to be effective. Anything above 50ms will cause a lag between the headset movement and virtual reality environment changes.

…

Eye tracking technology is still maturing, however, it may be one of the most important missing pieces to complete the virtual reality full immersion puzzle... Without eye tracking, everything remains in focus as you move your eyes – but not your head – around a scene, thus causing a greater likeliness of simulation sickness.

Aside from head tracking and eye tracking, VR also requires motion tracking. One of the big challenges for VR system developers has been to enable VR users to move around freely without being tethered by cords.

Computer Processing Power. Achieving all the requirements described above puts enormous demand on computer processing. As iQ by Intel describes it,

“When you are trying to do 90 frames per second, stereo 3D, over-1080p resolution, and have a very wide field of view at the same time? It’s just a lot of things to stack on top of one another,” Luckey explained.

While a computer runs these extreme graphics, it also processes data from the movement of the user’s head and body (picked up by a camera sensor) and 3D audio that adjusts to the user’s movement. That’s all while processing input from controllers, too.

The result is indeed an immersive game, but it is taxing for a computer. And if users don’t have a PC that’s up to the task, the picture jumps around, resulting in simulation sickness.

Complications

VR systems are complex bundles of different technology components. For VR systems to be successful, each and every component must achieve a high level of quality. However, each technology is complex in and of itself and is on a different path of development. Moreover, there are tensions across the different technologies that make it difficult to simultaneously optimize every component. For example, a VR headset must contain a truckload of technology, but at the same time, it must be lightweight, so as not to put pressure on user’s necks. In his review of Magic Leap, “Magic Leap is a Tragic Heap,” Palmer Lucky describes how Magic Leap addressed this tension (see Figure 9):

Figure 9: Magic Leap Lightpack

 

They call it the “Lightpack”. It is basically the guts of a tablet computer in an oversized hockey puck that you wear on your belt. This is the best part of the device by far, A+! I would have expected Magic Leap to do the fashionable thing and throw all their render hardware and battery power on the headset itself for looks, but some group of sane people appear to have recognized that putting your heaviest components on the most weight sensitive part of your body is a bad idea if you want people to actually wear your product for any period of time – this is a longer topic for another day, but the data shows that you need to be BRUTAL when it comes to reducing HMD weight. This approach also allows them to use much more powerful chips than they could feasibly cram into a head worn device.

Potential Solutions

My book, Winning the Hardware-Software Game, provides a full analysis of this type of situation for all different forms of technology systems, with lots of potential solutions.

At this point, the biggest deterrent to adoption of VR systems – by both users and content providers – seems to be that the hardware has not yet been sufficiently developed so as to create a true value proposition for users. Nevertheless, actions that might speed VR system adoption include the following.

Standards. Establish standards to avoid fragmentation. Easier said than done, especially with nascent technologies. Individual companies are pursuing their own technologies for a reason – they want to become the de facto standard, so they’re generally reluctant to agree to a different standard than the one they’re pursuing. Also, while establishing standards might help the process in the short run, it might end up locking in what turns out to be a substandard version of the technology for the long run.

Coalitions. VR hardware developers can create coalitions of content providers for their systems, that is, build up an ecosystem. Coalition members can then work together to jointly develop hardware and software products that end up in the hands of consumers sooner than otherwise. Users are more like to adopt systems with developing ecosystems earlier than they would if the coalitions didn’t exist. So establishing coalitions is a good way to kick-start adoption. Magic Leap is doing precisely this. Partners include the NBA and Star Wars owner, LucasFilm.

Subsidize Provision of Content. Platform developers can encourage content providers to develop more content for their platforms by decreasing content providers’ costs of development. Developer tool kits, APIs, training sessions, etc. all decrease content development costs and thus encourage the supply of content.

Subsidize Cross-Platform Development. Costs of developing content are high because content providers wishing to sell their content on different platforms have to separately tailor their content to meet each system’s specifications. Developers can encourage content providers to supply more content by decreasing the costs of porting content across systems. Unity Technologies helps companies do just this. From Sequoia Capital,

Unity tackles the hard, tedious work of platform management, so that game developers don’t have to. The idea is to build tools and systems so that game developers can work on making fantastic game play, while Unity does the all the brutal cross-device and cross-platform optimization.

Build on Current Use Cases. VR has been successfully used in a number of applications, including training, real estate and architecture, and therapy. Platform developers can build on these use cases in depth and/or breadth to generate more value and speed the pace of system adoption.