Cinematic virtual reality

Cinematic virtual reality (Cine-VR) is an immersive experience where the audience can look around in 360 degrees while hearing spatialized audio specifically designed to reinforce the belief that the audience is actually in the virtual environment rather than watching it on a two-dimensional screen.^[1] Cine-VR is different from traditional Virtual Reality which uses computer generated worlds and characters more akin to interactive gaming engines, while cine-VR uses live images captured thorough a camera which makes it more like film.^[2]

When storytellers began working in cine-VR, they applied many of the same cinematic narrative rules, but the technology demonstrated that VR can offer different possibilities that go beyond "traditional" cinema which will require new techniques and practices.^[3] Harrison Weber, journalist of Venturebeat, described cine-VR like this: "It's a lot like film, only it puts the audience inside your story. With it, you can create entire worlds for your audience but none of the original rules of cinema apply. How do you create your art when all of your tools have changed?"^[4]

The Human Interface Technology (HIT) Lab at the University of Canterbury differentiates cine-VR from other content created with 360-degree cameras based on the content, likening the prefix "cinematic" to that of "narrative". The HIT Lab requires cine-VR to be "narrative-based, instead of purely for novelty, entertainment, exploration, etc."; the cine-VR experience can be a drama, a documentary, or a hybrid as long as the story contains a beginning, a middle, and an end.^[5] According to Ohio University's Game Research and Immersive Design (GRID) Lab, a cine-VR project differentiates itself from 360-degree video by using cinematic production techniques such as lighting design, sound design, scenic design, and blocking techniques (the latter two in the case of dramatic work).^[6]

The concepts of "immersion" and "presence" are central to cine-VR.^[7] The term presence is defined as "a sense of being there"^[8] and described as "a feeling of actually being on location in a story rather than experiencing it from the outside".^[9] Scholar Christian Roth differentiates immersion from presence by defining immersion as an objective criterion which depends on hardware and software, while presence is defined as the more subjective, psychological sense of being in the environment, and mainly influenced by the content of that world (e.g. story, characters, and location).^[10] Immersion could be seen as a quality of the medium, in this case a cine-VR experience, while presence is a characteristic of the user experience; hence, higher immersion may lead to or result in deeper presence.^[11] Immersion has objective components that can be advanced by technical considerations like image quality and sound quality, while presence is affected by individual users' subjective variations but is aided by the technical aspects that foster immersion.^[12]

Cine-VR provides a more photorealistic user experience than traditional virtual reality, but current technology does not allow the audience to move around in video. In some ways, cine-VR is a trade-off, as fully computer-generated VR looks less realistic than cine-VR but is more interactive.^[12] The ability to look around inside of a virtual reality space is known as three degrees of freedom (3 DOF), while being able to move around inside a virtual environment is known as six degrees of freedom (6 DOF). 6 DOF should ideally enhance the user's sense of presence.^[7] The 3 DOF of cine-VR are defined as yaw (rotating your head left or right), pitch (tilting your head up or down) and roll (tilting your head on its axis upside down or right-side up).^[3] Since experiencing a story using 3 DOF is quite different from watching a traditional film, television show or stage play, storytellers have recognized a need to develop a new creative language for cine-VR.^[13] The key element that differentiates cinema and cinematic VR is the new role of the audience. Technologically speaking, this requires the storyteller to embrace the concept of immersion.^[3]

With the ability to use 3 DOF, the cine-VR audience can freely choose the viewing direction when they experience the story. Therefore, traditional filmmaking techniques for guiding the viewers' attention cannot be used: techniques such as panning the camera or cutting to a close-up shot are no longer available to the filmmaker; instead it is the viewer who decides where to look.^[14] Subsequently, in cine-VR, the storyteller has to rely more on lighting, sound design, and how the characters and sets are arranged to best tell the story.^[13] Famous filmmakers have been attempting to do this at least since the mid-2010s when Kathryn Bigelow directed the cine-VR piece The Protectors (2016), Doug Liman directed Invisible (2017) and Alejandro Gonzalez Iñárritu, debuted Carne y Arena / Flesh & Sand at the Cannes Film Festival in 2017.^[2]

A variety of cameras can be used to create cine-VR images, including traditional cinema cameras in conjunction with a panorama tripod head. Most commonly 360° cameras are used, allowing the storyteller to capture the entire 360° space at one time. 360° cameras use multiple lens combinations to capture all portions of the 360° image simultaneously. Those disparate images are then combined into one 360° panoramic image using a process called "stitching". If a single lens faces in one direction, the image is referred to as monoscopic. If two lenses are used for a single direction, the image is referred to as stereoscopic.^[15]

Stereoscopic imaging creates a 3D effect. This technique leverages the parallax difference between multiple lenses to achieve the illusion of depth. Stereoscopic content is generally contained within one media file with the images stacked above and below each other or in a side-by side-fashion.^[7]

Ambisonics is a method for recording, mixing and playing 360-degree audio. It was invented in the 1970s but was never commercially adopted until the development of the virtual reality industry (including cine-VR) which requires 360° audio to match with the 360° images.^[16] Audio designer Simon Goodwin describes ambisonics, as "a generalized way of representing a soundfield—the distribution of sounds from all directions around a listener."^[17] Ambisonic audio recreates the soundfield spherically and is uniquely suited for VR applications because it provides motion-tracked variations of audio signals and enables sounds to be positioned anywhere around a user—up/down, front/back, left/right. Properly implemented, Ambisonic audio allows users to move their heads and bodies around in the soundfield just as they might turn to look for the source of a sound in real life. As users glance around, the headset uses motion tracking to alter sound direction and quality.^[7]

Ambisonic techniques are needed which guide the attention of the cine-VR spectator toward important visual information in the scene. Attention guidance using ambisonics improves the general viewing experience, since viewers will be less afraid to miss something when watching a cine-VR story.^[14]  A level of aural realism is achievable by combining ambisonic recordings of environments with dialogue captured through a traditional microphone (usually a lavalier) along with added sound effects generated through foley work.  Ambisonic audio, in combination with traditional microphones and sound effects, plays an important in creating a sense of immersion for the user experience.^[18]

Cine-VR should ideally be played on a head mounted display (or HMD) with headphones. While in such a headset, most surrounding distractions are visually blocked out. HMDs have built-in hardware sensors called gyroscopes and accelerometers to move the images in concert with the audience moving their head. Gyroscopes track how much something is tilting and help to smooth the graphical playback to prevent videos from shaking. Accelerometers measure the actual movement in space. The combination of these two can precisely track the device's position and orientation. Along with optical or infrared tracking, gyroscopes and accelerometers are integral parts of the VR headsets' tracking capabilities and increase immersion.^[19] In contrast with the other mediums, "immersion" is still the main experiential value that the experts remark in cine-VR.  This is achieved primarily through head mounted displays used to view its content.^[3]

According to John Bowditch, director of Ohio University's GRID Lab, "The VR industry is trending towards wireless HMDs due to consumer demand and ease of use; however, the overall quality does not currently match the performance of headsets wired to a PC. Some wireless headsets support wired-connections to computers (usually with a USB cable) for more processor extensive applications. Wireless headsets are generally less expensive because they do not require a PC to operate. Most wireless headsets can be initially configured with a smartphone and then run independently by downloading or streaming content through a Wi-Fi connection. Wireless headsets tend to be more comfortable, easier to transport, and work well for both seated and standing experiences. Most cine-VR playback is either seated or standing and will not require walking around. Swivel office chairs that rotate 360° with ease are our preferred furniture. However, any furniture that doesn't restrict your audience's movements is usable."^[20]

• Entertainment technology
• Extended reality
• Immersion (virtual reality)
• Metaverse
• On-set virtual production