Gerald Underwater
Teamwork augmented reality project (Hang Wu, Irene Li, Kiki Apple, Kristin Wu)
A composite view of the underwater world around Gerald (the little robot) and our studio space
Music: Haruomi Hosono
Reflection Essay
Hang Wu
I have been really into different kinds of digital media practices and enjoyed spending time working on practical projects while writing academic papers. The two things always talk to each other in different ways. Taking this course is also an extension of such interest to see how practices and theoretical thinking could inform each other and give a hybrid child. On the one hand, I have some experience in game production and programming languages and a strong interest in working on digital media arts. On the other hand, I also have a theoretical interest in new media and digital media as part of my research project. As my dissertation is concerned with new media and digital media technologies, one of the crucial research questions for me to explore is the integration of CGI or digital effects and live-action filming. The compositing of visual effects and live-action footage has been increasingly common in contemporary filmmaking and new media practices. Regardless of how problematic it may appear to be as a principle, an ideal seamless integration between the two worlds has always been a challenge for film technicians and media producers. As for me, a cinema and media study student, I am very interested in making sense of this phenomenon: How do we understand such compositing? How does it alter our understanding of reality, media, the mediated environment that is our society? And how do we situate humans in this media assemblage and in what mode of existence?
Yet AR is also different from effects-driven films or TV shows, as it requires a moving camera lens, and thus affords mobility that the latter does not often have. It is lighter, smaller, and somehow appears to be much more personalized. The emergence of AR cannot be divorced from a media environment in which the smartphone, with its great computing and processing power, plays a significant role. Other major actors in this environment include a telecommunicational network with increasingly higher bandwidth as well as social media platforms that are accessible on mobile devices. The smartphone-centered media environment in which AR thrives may help explain why some of the active promoters of AR include TikTok and Snapchat.
Because augmented reality (AR) affords a particular way of interacting with reality—by augmenting it—as its name implies, I think it offers a great opportunity for me to consider these questions I mentioned above. Some of the sketches that I made as well as our group project has explored the relationship between AR and reality. As for my sketches, I tried to place the spinning planets I made over Lake Michigan. I added the hollow and spooky sound of Saturn’s radio emissions from NASA to the work. Altogether, the sound and the plants reinforce a cosmological feeling of Lake Michigan, which is already embodied in its formation in the age of the continental glacier. Evoking a cosmological scale of the natural history, the sketch somehow turns Lake Michigan from an everyday space for a walk or a picnic into something different.
Gerald Underwater
As for our underwater final project “Gerald Underwater”, the four of us got together and did a short Q & A session to talk about the project, and we all touched on how our work transformed our sense and perception of everyday reality. I made a short video to present our AR work in one of the computer rooms of the MADD Center. We chose this place as MADD Center has become an essential space where we attended classes, gathered to discuss and work on collaborative projects. In addition, we were happy to see that the shimmering blue water, waving jellyfish, seaweeds, the whale, and Gerald that we made for our project give this space a very different aesthetic quality. An underwater computer room is a paradoxical place, and the paradox makes it a work of art. It reminds me of what Luhmann writes about art in Art as a Social System, a book that I was reading for my Japanese literature class around the same time we made the AR project. Luhmann writes about the paradox of an artwork is that it always points to what it does not say or represent, or what he calls “external effects to the cosmologically or socially grounded service functions of a given activity.” What is conveyed in this project is no longer merely the 3D models, the simulated water movements through the execution of code, or the computer room in an art center, but the overall effect generated by their assemblage and the world that this effect gestures toward.
In the initial stage of the final project, we, the pretty blender group members who want to make something that looks cool and beautiful with our AR toolkit, had a meeting to brainstorm ideas about the final project. Kiki mentioned that she wanted to focus on the theme of underwater, while Emily proposed to work on things over the cloud. I was excited to hear their proposals as I always think that the otherworldly feeling or the perception of isekai—to use a Japanese term—is exactly something that AR affords. As I happened to take a class on Ocean Media last year, in which we learned so much about marine life and the thermodynamics of ocean waves, I decided to join the underwater group. In our first group meeting, we spent most of the time discussing how to actualize an oceanic feeling. I suggested making some jellyfish 3D models, as seeing jellyfish has always been one of my favorite things to do at the Grand Aquarium in Hong Kong. The cover of Paul Roquet’s book Ambient Media: Japanese Atmospheres of Self on the post-1980s Japanese media in its neoliberal economy has also inspired me to think about the jellyfish-ocean coupling. Kristin made a beautiful jellyfish in Blender, but it was unfortunate that we were not able to export its colorful texture from Blender to Field. The music that I ended up making is a synthesis of songs from Hosono Haruomi’s album Mercuric Dance, including “Sunnyside of the Water” and “Mercuric Dance” which I learned from Roquet’s book, and the sounds of blue whales from deep underwater. The selection of music also inspired us to make corresponding 3D models that echo the sound—Kristin made a giant whale, which turned out to be amazing in the final presentation.
Shader and the Elemental Media
Because of her familiarity with the shader, Irene was mainly responsible for making the water and its gleam. I believe that the use of the shader to generate a deep-sea space with sunbeams, water ripples and bubbles is an essential part of our project. As for the specific steps, we first made an oval-shaped skybox so Irene can apply this specific water shader to this skybox, and then we fixed the camera of the phone inside the skybox. So, no matter camera changes position or how the person who holds the camera moves around, what is presented in the XRViewer would always be the underwater space. To a certain extent, the skybox that we made functions as a filter, which adds special effects to what has been captured by the camera from the physical reality. However, it also operates on a totally different level, as it simulates a 3D space that overlaps and intersects with the real world through the camera and the computation of the mobile device.
The process of making generating the effects of water through a shader also inspires me to think more about elemental media, or natural elements such as water, fire, wood, metal as a medium, as proposed by media environment scholars. I am intrigued by the close affinity between shader as a visual effects technology and natural elements as a cultural technique. In addition to our project that experiments with water, in one of the classes, Professor Marc Downie also showed us how to apply a fire shader to a dancing body. I feel that the rise of the elemental media is inextricably linked to visual effects techniques such as shader and particle systems. Questions that are worth thinking about further are: does this affinity generate a different structure of feeling and subject formation? And what kind of forces and actions the elemental media brings in our society structured largely by global capitalism and nation-state?
Who is Gerald?
We decided to bring Gerald back at a later stage when we started to think about the kind of interactive feature we wanted to implement into the project. Gerald’s aesthetic features, including its simple color scheme and rudimentary 3D modeling, do not match perfectly with other parts of the project—but it did not stop everyone (including our teammates and the rest of the class) from being the fans of Gerald, as Gerald is on the other hand very refined in terms of its interactive features. Our class had a Gerald cult—Non even brought Gerald stickers to share with everyone in the last class. It is Kiki’s coding that allowed Gerald to play the hide and seek game with us. Gerald moves swiftly. Sometimes they look at you. And in the next moment, they decide to chase after you if you happen to pick up their favorite snacks.
We always got people asking: who is Gerald? We all burst into laughter hearing this question. Sometimes we quoted Marc and answered, Gerald is Kiki. Sometimes we also said, Gerald is everything. In our last working session, Kristin and Kiki photoshopped Gerald on the lake, Gerald above the clouds being worshipped, and Gerald on the beach. We all had a lot of fun playing with Gerald. It reminded me of my favorite game character Mario. It often begins with a character, a tiny idea, or a basic feeling of the ocean, and things started to be built around it as people are having fun making something together. And it might end up with multimedia universes.