It is 2020, and extended reality has not yet extended all the way into the public sphere. Talk of the technology reverberates in small echo chambers, incorporating Silicon Valley technologists, gamers, advertisers, and consulting firms. The stakeholders for immersive journalism make up an even smaller community. In the United States, only the biggest news organizations have been flush with the expendable capital to invest in AR/VR research, so they are the ones making the case studies from Chapter 2. Still, XR experiences are not yet a substantive part of public life — the industry is trapped in the thick brush of novelty, struggling to push beyond the wow factor of a first VR experience or the social virality of AR filters on Snapchat and Instagram. Are we in the midst of a long winter for extended reality, before a lush spring of proliferation? Despite slow growth, Mark Zuckerberg, one of XR's more impatient groundhogs, has not seen his shadow. An early spring is coming. “THE TIME IS NOW,” he and the rest of Facebook/Oculus team screamed from the keynote stage (and printed on the staircases) at Oculus Connect 6 in 2019.
Facebook CEO Mark Zuckerberg delivering a keynote address at OC6, on September 25, 2019. Photo taken by the author.
Staircase at the San Jose Convention Center for OC6. Photo taken by the author.
Effusive technologists are pushing the envelope in the XR industry... expediting the construction of “the next computing platform” (pictured above). They know that AR and VR are stuck in their silos because the infrastructure for public consumption is not quite ready. The Internet as we know it today is not built to host immersive media; the form factor is too clunky, the price is too high anyway; regulation lags behind; 5G isn't quite there; privacy red flags persist; and Black Mirror is still a main reference point in the public imagination. None of these impediments are slowing down Zuckerberg and others' sense of urgency: the wheels are very much in motion to refine the infrastructure for XR-mediated public spheres.
Right now, technology companies are using cameras on iPhones, drones, and CCTVs to capture every street corner, building, home, and park in the world. The goal: use computer vision AI to triangulate the disparate images hyper-accurate 3D maps of the world, so camera devices can recognize their surroundings and position themselves within it. Spatial computing, as this process is called, enables computers to perceive, retain, and manipulate physical spaces with three-dimensional fidelity. It transforms the entire built and natural environment into a potential interface for human-digital interaction [1]. Indeed, this infrastructure enables the localization affordance of augmented reality. Every place, plane, body, or object becomes a canvas for AR, visible through the smartphones of today and the mixed reality glasses of tomorrow. Spatial computing is well under construction, largely without the knowledge, consent, or understanding of users and regulators.
Benjamin Bratton's framework of “The Stack” helps to understand the layers of this infrastructure: User, Interface, Address, City, Cloud, and Earth [2]. Users have smartphones or mixed reality glasses (like the Magic Leap One, Microsoft HoloLens, or Google Glass), which activate the Interface. The user interface is manipulated with hand/body gestures, voice commands, or even eye-tracking within the glasses. Through the interface, users encounter the Addresses, which are the digital data tagged to specific locations in the larger 3D map of the City layer. The city leans on fast 5G network speeds to connect to the AR Cloud layer, where all the digital content is stored, processed, and drawn from. The cloud layer requires massive data banks, which thereby drains the Earth layer's energy.
The video (on the right) once again showcases the Interface layer of spatial computing, as exemplified by a demo of the HoloLens 2. Note the seamless hand gestures and voice commands. It is the latest expression of XR's historical trend of the disappearing interface, discussed back in Chapter 1.
The tweeted video on the right demonstrates the Addresses layer of spatial computing at its finest. The AR experience was created by Scape Technologies, the London-based mapping company. Through their ScapeKit SDK, the smartphone camera recognizes those sidewalks and buildings, which have been demarcated as addresses for augmented content. Note how the flying cars turn the street corner with such realistic ease (0:27)... that one second flexes the fidelity of the Scape map of London.
In my understanding, these two videos depict the state of the art in spatial computing (at least what's publicly available). Technology titans have started building the basic Interfaces, compiled a few Addresses of digital content, and formed partnerships with telecommunication companies like AT&T to power the next step: the Cloud layer.
https://youtu.be/uIHPPtPBgHk?t=64
https://twitter.com/TweetEdMiller/status/1189255383879208960
Whether we like it or not, spatial computing is critical to hosting XR journalism experiences. On top of this engineering infrastructure, the Magic Leap One, Microsoft HoloLens, Google Glass, Oculus headsets and eventual Apple wearables need to be networked together, so XR experiences can be easily passed between them and accessed from anywhere and on any hardware. In other words, the missing piece is the Cloud layer: I refer to it as the spatial web, and it is where presence-driven VR experiences, contextualized AR content, and spatial storytelling will live. It follows that the way in which the spatial web is stitched will play a huge role in how people create and interact with immersive journalism in the public sphere. So, yes, the time is now. **The time is now to act decisively and build ethical, interoperable, organized, inclusive, and privacy-oriented infrastructure of the spatial web. Before we go too much further, this chapter goes behind the Silicon Valley curtain and chronicles how technology leaders are currently conceptualizing and constructing the spatial web. I first tell the story of those who are attempting to control XR's future; and then offer a path forward to wrest control away from them.
Google provides Visual Positioning Services in its Maps and Mark applications, allowing for navigation directions and digital art respectively to be localized in the world with AR [3]. Apple offers visitors to their retail outlets the chance to experience augmented "Art Walks" through major cities [4]. Microsoft furnishes datasets of Spatial Anchors to pin digital content to locations in the physical world, powered by its Azure cloud [5]. Florida-based Magic Leap is developing an entire “Magicverse” of digital layers for connected entertainment, healthcare, and more [6]. There is of course Scape Technologies, the company from the above video that mapped all of London to the centimeter and is planning 100 more cities. But in February 2020, Scape was acquired by Facebook, in an unheralded purchase presumably to aid Facebook's new LiveMaps project. In a statement about the acquisition in February 2020, a Facebook spokesperson said: “We acquire smaller tech companies from time to time. We don’t always discuss our plans” [7]. Well, watch the LiveMaps video below, and you will know exactly what Facebook's plans are and see where the video demo from the London street fits in. Lastly, Niantic, the gaming company behind Ingress, Pokémon Go, and Harry Potter: Wizards Unite, recently acquired imaging company 6D.ai. In a post from March 2020, CEO John Hanke wrote: "Together, we’re building a dynamic, 3D map of the world so we can enable new kinds of planet-scale AR experiences"[8]. Big technology companies are each planting their flags on the digital metaverse that will house extended reality content, and this list is not even exhaustive.
.png)
Google's Live View AR in navigation.
Apple art walks begin at Apple Stores and allow users to encounter digital artwork in the built environment.
Microsoft graphic explaining its Azure Spatial Anchors.
Magic Leap graphic about the Magicverse.
Visualization of an augmented city powered by Scape Technologies, now owned by Facebook.
Niantic visualization of planet-scale AR.
I watched this video from the audience at the Oculus Connect 6 keynote. Everybody clapped. I was in shock.
Facebook, Apple, Microsoft, Google, and Niantic depict the use cases of spatial computing as playful, innocent, or purely efficient: games like Pokémon Go and Minecraft Earth, Apple's art walks through New York City, and AR navigation through Google Maps try to make this digitally remapped planet and powerful spatial computing look like something you would want, even need. This marketing masks fundamental dangers. Within this infrastructure lies the foundation for massive surveillance and pernicious data collection, and this is where we need to be tremendously careful.
/cdn.vox-cdn.com/uploads/chorus_image/image/64956786/googlemaps.0.jpg){kind=link}
