Movers and Shakers


There are at least 600 monuments to Christopher Columbus around the world [1]. For decades, activists have vandalized, protested, and lobbied against the physical prevalence and permanence of this figure: both an explorer and a perpetrator of genocide of indigenous people. Immersive artist Glenn Cantave found a new approach to fight back against this entrenched narrative in public space: augmented reality.

Cantave and his nonprofit, Movers and Shakers NYC, create AR monuments: digital 3D statues of heroes from marginalized communities. Colin Kaepernick, Serena Williams, Simon Bolivar: you can find these digital monuments pinned to specific locations throughout New York City and accessible via your smartphone camera. The AR monuments draw attention away from Columbus and juxtapose him against contemporary heroes [2]. This AR project animates oppressed archives and highlights crucially different perspectives.

[AR Colin Kaepernick placed in NYC subway. Click to enlarge Photo from Movers and Shakers.](https://lh4.googleusercontent.com/0QgQKeJxQqT0AO0Dib9TxgTY1wDLv2e6YazfNAUXggY5sQXO0kwMox3_EkjNTmrcmQCJ-HWlAkiMi5DPYe085Yv5jP8_Em2_iMS34T7SfvqFknkKLcaqm92P6oPEQDFk2BnWAewPUTrnjntwkQ)

AR Colin Kaepernick placed in NYC subway. Click to enlarge Photo from Movers and Shakers.

Cantave shared his work in a TED Talk about augmented reality activism: he sees these custom AR projects as a tool to embed "stories in public spaces that need to be told... You don't need permission from the government to put up a monument or make a statement. You can just do it" [4].

Before having seen Cantave's work, I was working with the Brown Daily Herald as the collegiate paper's first immersive editor. I started the team hoping to experiment with virtual and augmented reality technology for journalism. In October 2019, one editor proposed a story covering the latest vandalism of the Columbus statue in Providence — vandals had covered the monument in red paint, sprayed it with block orange letters screaming "GENOCIDE" and left a wooden sign compelling the city to "Stop Celebrating Genocide." In response, Providence city officials rushed to wash down the paint and remove almost all remnants of the event. Prior to that, however, some local newspapers and TV networks captured a photograph of the vandalism for their own reporting.

Our immersive team visited the Columbus Statue after the paint had been washed, and scanned it as a 3D model using a technique known as photogrammetry (more in Chapter 2). We took hundreds of photographs around the object and used modeling software to assemble the disparate angles into a fully textured 3D replica of the monument (right). If you look closely, spots of orange graffiti and swaths of red paint remain.

We studied the photographs from the Providence Journal, which were captured right in the nick of time before the city hosed down the statue. Relying on the ProJo's photos, we saw an opportunity to preserve the paint in a way that would maintain the spirit of the protest. We decided to digitally re-vandalize the statue, re-painting the red paint and re-writing the orange graffiti in Blender. We wrestled with the ethical implications of this re-creation. The digital overlay is not 100% accurate, and "GENOCIDE" had to be written with my own wobbly handwriting. But we landed on a textual explanation of our effort as sufficient to rationalize our work. The goal was to properly re-create the protest with the highest fidelity we could, to show the story, even after the city had erased it. It was not intended as an act of solidarity with the protest or a discrete political statement. We published the story online with embedded augmented reality capabilities: anyone with an iPhone or Android can place the 3D models of the statues in their space, to walk around, interact with, and understand [4].

Augmented reality restored the ephemeral paint and provided an archival documentation of the event for public knowledge. The vandalism was an act of speech and storytelling, an attempt to properly contextualize a person by shedding light on the darker elements of our complicated history. It counteracted the dominant narrative, so the city erased it in order to maintain order and protect the public perception of the statue. Augmented reality fought back.

Original Columbus statue scan. On mobile, click the AR logo in the corner to view it in your space.

Original Columbus statue scan. On mobile, click the AR logo in the corner to view it in your space.

Digitally re-painted vandalism.

Digitally re-painted vandalism.

These projects happened independently. Movers and Shakers' work with AR monuments and our Herald Columbus piece, however, agree on an optimistic capability of this technology to reveal concealed perspectives, contextualize history, and impact public spaces. More on those distinct AR characteristics in Chapter 2. These monument projects represent a best path forward for immersive media, but the tools could just as well be deployed unthinkingly and/or maliciously. Augmented and virtual reality are growing up; their identities and priorities are in flux. These extended reality technologies (XR) are as of yet undetermined, unstandardized, and untethered to any one private company's vision. There is tremendous work to be done in pointing their trajectory, setting ethical guardrails, thinking critically about their potential impact, and moderating their use in public life.

This thesis explores the intersection of three concepts: extended reality (XR), journalism, and public spheres. It belongs to a moment of time — 2020 — when XR technologies are emerging, making their way into journalistic practice, and therefore altering the structures of public spheres. To make sense of this story, we need first to define the scope and establish the stakes for each of these concepts.

Extended Reality


Augmented and virtual reality are slowly permeating our daily lives. AR overlays digital content onto the physical world. VR immerses you in a completely digital environment. Extended reality (XR) is an umbrella term encompassing both AR and VR.

Graphic from DARQ studio.

Graphic from DARQ studio.

With each passing year, the technology advances in computing power and consumer popularity. Big Silicon Valley companies are doubling down on AR and wielding their influence to drive adoption. Pick up your Android or Apple smartphone today. It is automatically equipped with ARCore or ARKit, respectively: integrated software tools that empower the phone camera to interpret physical world planes and image markers as canvases for digital content like 3D models. Social media platforms are updated with AR capabilities: Spark AR powers Facebook and Instagram effects, while branded Lenses pervade Snapchat. Amazon enables you to envision potential purchases in your home with AR. Googling any animal yields a 3D model at the top of the search results, so you can plop a photorealistic animated bear on your kitchen table. On the enterprise side, high-end mixed reality glasses like the Microsoft HoloLens and Magic Leap One enable interactive 3D visualizations and augmented workflows. Anyone can create: AR authoring tools like Apple Reality Composer and Adobe Aero are free and intuitive. Everyone with a smartphone and a software update — over 1 billion devices — has AR experiences and authoring tools at their fingertips.

As for VR, the long-anticipated technology is finally finding a market [5]. Web browsers and social platforms include plug-ins for 360 viewing, so anyone can drag around a spherical image or video to engage with an immersive VR story. A $10 Google cardboard wearable overtakes your field of view, or a $400 Oculus headset transports you to another world. VR headsets pop up at museums, art galleries, theme parks, and are sold with gaming consoles; they have found their greatest utility and proliferation in job training simulations for enterprises. The list of gaming, narrative, and training experiences expands exponentially as the creating tools become more accessible and production costs come down. What's more, with their newly integrated (albeit lightweight) AR/VR capabilities, the digital appendages we call smartphones are potent tools for immersion.

This is the present in 2020. The debate surges about projected futures for this technology, overwhelmingly between those evaluating it under a binary of either good or bad. Early adopters exude excitement, as Facebook (and its VR brand, Oculus), Snapchat, Microsoft, and Apple earn rounds of applause at their developers' conference keynotes. These cheers of praise get written into publications like VRScout, Next Reality, and Wired that cover the constant innovations in the space. Futurists get wind of the progress and spill ink into self-proclaimed prophetic books, filled with install base metrics, market projections, and the future simple: extended reality will change the world. TED Talks bolster the buzz by planting seeds like VR as the "ultimate empathy machine" [6]. Excitement for extended reality generally echoes in this small but ever-expanding community, as more industries and organizations start to see the utility and efficiency XR offers. The product of this echo chamber is a utopian vision of XR-mediated public life, focusing on all the potential uses and beneficial affordances of this technology.

In this vision, augmented and virtual reality will help us lead better lives. Qualcomm, the chip maker powering many XR devices today, shouts their conviction that XR will be the next mobile computing platform; coupled with 5G cellular speeds, they argue, the possibilities for high fidelity immersive experiences are limitless [7]. XR wearables will finally free digital information from the prison of the 5-inch screen as it diffuses into real locations in the physical world, localized into massive city-scale 3D maps. Look at a restaurant window and see Yelp ratings populate; walk down Broadway in NYC and see 3D advertisements for upcoming shows pop out of the facades of the theaters; encounter digital monuments of women and people of color who deserve to be celebrated. Mixed reality glasses will serve as customizable windows to this spatial Internet, with which we interact via natural bodily gestures, voice commands, and even eye-movements. Workers from car factories to hospitals will increase productivity with 3D visualizations and VR training. Headset owners will create and nurture Social VR worlds: boundless digital sandboxes where you could express yourself with your avatar and defy reality in plane-flying games or 3D art rooms with your other headset-owning friends. Optimistic futurists hail this moment as a chance to reverse the mistakes made by Web 2.0 [8]: the spatial Internet could be open source and decentralized, cleansed of the power structures like Facebook and Google that control the web today. As you may have picked up: Web 3.0 is already dominated and dictated by the familiar Silicon Valley power brokers. More on the infrastructure of the spatial web in Chapter 3.