Augmented Curiosities: Virtual Play in African Pasts and Futures is an immersive exhibition that places African material culture in the hands of its visitors. As an Innovator-in-Residence for Media Technology & Innovation, I developed the project through collaborations with the Northwestern University Library, most specifically the Melville J. Herskovits Library of African Studies, and Northwestern IT. Augmented Curiosities is designed to promote the Herskovits’ incredible object collection and provoke conversations about the use of Virtual Reality (VR) and Augmented Reality (AR) within the storytelling strategies of museums.

I’m an archaeologist and a curator. Within these disciplines, I use material-based methods to explore my interests in expressive and tangible culture as a force of preservation – especially within cultures that do not rely upon the written record. Archaeologists understand that objects and artifacts often embody characteristics of the people and processes that created them. Craftspeople, artists and producers of many practices embed their unique perspectives within the fabric of their works. The energy of an individual, community or landscape can often be gleaned from the materials it produces. Because of this, we curators often use objects to tell stories of the human past and widen an audience’s capacity for imagination. I’m particularly interested in widening imaginations about African pasts and futures. And curation provides me intellectual space to produce scholarship as an artist within an archaeological practice. I work to engage that artistry through experimentation with technology and digital media. Augmented Curiosities is my most recent effort towards that end.

For this project, I sought to find ways to place fragile, valuable or otherwise inaccessible objects from the Herskovits Library of African Studies collection into the hands of visitors. To accomplish this, our team integrated Virtual Reality and Augmented Reality experiences within an exhibition, that allowed visitors to intimately engage the objects. We used photogrammetry and 3D modeling to create a digital replica of each object and the exhibition cabinet in which they are housed. We then made those replicas accessible through the development of a VR application in Unity Game Engine and a web AR experience through the 8thwall platform. This curatorial framework allowed me to express historical, archaeological and anthropological data as time, space and experience. The VR and AR content delivery strategies offer visitors an immersive experience in which they may virtually remove the objects out of the glass-paned exhibition cabinet and play with them. Through this we created a product that was both educational and entertaining – simultaneously exhibiting emerging technologies and relatively hidden objects.

Northwestern University Dean of Libraries, Xuemao Wang, trying out the Augmented Reality Experience

Augmented Curiosities was partly inspired by my interest in affective curation. I believe objects possess energy that humans transform into meaning. We deploy our personalities, emotions and unique lived experiences to make sense of our physical surroundings. I want to curate this phenomenon. Rather than highlighting geographic origins or manufacture dates, I am far more interested in allowing a visitor to have experiences with material culture. I sought to exhibit the feeling of the objects as opposed to prescribing their meanings. To archive this within the exhibition, we produced video interviews from Northwestern University community members that contextualize each object within emotional, political, spiritual and familial landscapes. Interviewees were prompted to respond to the question: “What does this object mean to you and how does it make you feel?” The videos of their testimonies create an atmosphere of intimacy as visitors explore within the VR experience. Affective curation allows me to invert some of the power dynamics of exhibition development, delegating parts of the knowledge-production process to the communities that the exhibition engages. I am eager to further develop this theory and methodology throughout my career.

This experience affirmed for me the expansive opportunities offered by digital curation. The Augmented Curiosities project facilitated the production of a new archive – consisting of 3D models, video, audio and literature. The project created and consolidated a dynamic range of new and existing resources that support cursory browsing or rabbit-hole deep dives into the objects, communities and research that constitute the exhibition. Following the guidance of Esmeralda Kale, the curator of the Herskovits Library of African Studies, I created an Augmented Curiosities LibGuide which preserves the exhibition within the Northwestern University Library database. It is my hope that this rootedness within the digital infrastructure of the Library will encourage further inclusion of digital resources, such as 3D models and expert testimonies, within the University’s digital repositories.

Augmented Curiosities is the product of ambitious ideation and incredible institutional support. An expanse of Northwestern University units contributed time, expertise and funding to produce this unprecedented exhibition. I am grateful and indebted to the talented group of developers, engineers, scholars and technicians that believed in my idea and assisted its creation. Augmented Curiosities is a testament to the worthiness of interdisciplinary research and the ease in which meaningful cross-campus collaboration can occur at Northwestern University.

Personally, this project has been a tremendous success. I tested hypotheses, gained methodological training, developed theory and shared a product of the process with the Northwestern University community – an exemplary graduate student research experience that I’m extremely grateful for. Ultimately, it is my hope that Augmented Curiosities makes evident the power of immersive visualization techniques and encourages visitors to critically reflect upon the value proposition of exhibitions and the museum industry.

In our mission to promote the use of emerging technologies for the Northwestern University community, Media and Technology Innovation (MTI) partnered with the Anthropology Department to bring innovation to the forefront of the classroom. Anthropology Doctoral Students Craig Stevens and Eli Kuto, alongside Nate Bartlett and the MTI team, joined forces to create an unforgettable lab experience for students in the Fall 2023 term Archaeology: Unearthing History (214-0-1) course.

Powered by the whirr of drone propellers and the magic of 3D photogrammetry, we embarked on a mission to unearth the history of our very own campus landscape. In the hands of intrepid anthropology students, our Unmanned Aerial Systems (UAS) weren’t just delivering pizza, they were investigating the history of Northwestern’s iconic Lakefill and capturing the intricate tapestry of the landscape below.

Dr. Rozensweig (Anthropology), Nate Bartlett (MTI), Eli Kuto (Anthropology), Craig Stevens (MTI), and ANTH 214 lab section participants waving for an aerial drone photograph.

With the leadership of course instructor Dr. Melissa Rozensweig, the MTI team set out to explore how to integrate the technology in ways that were safe and accessible for students while also figuring out the logisitics of doing this kind of activity with a large-enrollment class. Through consultation and collaboration with the NU Environmental Health and Safety Department (EHS), an extra layer of security was added to our high-flying endeavors.

Our team sought to create a hands-on exploration that highlighted the transformative power of technology in understanding landscapes. To accomplish this, participating students operated a drone in supervised missions, flying gridded patterns over the Lakefill landscape. As preparation, each student read and acknowledged an MTI-crafted safety acknowledgment form – provoking a touch of excitement and responsibility, ensuring students were mentally prepared for their airborne missions.

Over four days, 77 Northwestern students launched a drone from the North lawn of the Norris University Center, investigating and documenting the Northwestern University Lakefill from hundreds of feet above the ground. Their mission was to unravel the historic formation of the campus landscape, using cutting-edge technology to explore the past from an immersive bird’s-eye view.

Guided by key questions on visibility, proximity, accessibility, location, context, and significance, students analyzed a 3D model of the Lakefill, which was made possible by our week-long aerial data collection. The aerial perspective prompted students to contemplate changes, both subtle and profound, in the Northwestern Lakefill and Lagoon. Students compared pre-Lakefill maps with present-day 3D models, witnessing vanished shorelines and the emergence of an innovative architectural marvel. They traced the veins of paths and sidewalks, wondering about the students of the past that once traversed them. This wasn’t just about studying history – it was about feeling its pulse. More than a demonstration of technology, we collaboratively designed a lesson in human-environment relationships and the impact of anthropogenic factors on landscapes. From above, the familiar campus transformed into a living, breathing timeline. And the sculpted earth revealed the fingerprints of human intervention.

A screenshot of the Lakefill and Lagoon 3D model in the DroneDeploy web interface.

The impact of this lab experience went beyond the classroom. Students didn’t just analyze the Lakefill – they reported on it, creating detailed accounts of its formation and evolution. For homework, students produced a landscape formation report based on their analysis of the 3D models, providing a unique and insightful perspective on the Northwestern Lakefill. Check out some of their enlightening takeaways:

Student A: “From the aerial and 3D view of the Lakefill and lagoon, a lot of the Lakefill area and a significant chunk of campus can be seen through different angles. The aerial view is significant as it can help you judge distance and acreage, elevation levels, plant heights, etc”.

Student B: “The 3D model of the Lakefill emphasizes the man-made lagoon, as the model does not include the entirety of the Lakefill. You gain a clear picture of the density of the foliage, the landscape design of the land near the lagoon, and information on the buildings closest to the lagoon. The aerial view emphasizes the shape of the Lagoon, and its sheer size, marking it as a dominant part of the Lakefill”

Student C: “The aerial view of the landscape provides a detailed overhead view. Using an aerial view displays visibility that is not possible to see from field walking or ground survey. The 3D model centers around the Lakefill and provides imaging of the surrounding landscape”.

Our team was delighted by the first-hand and original analyses of our campus that the students contributed. The lab experience became a pedagogical innovation for Northwestern, showcasing the potential of technology to make education entertaining and impactful.

Lab participants spotting the drone after the instructional launch.

The collaboration with the Anthropology department is set to continue, with plans in the works to regularly offer this lab activity for the Archaeology: Unearthing History course. Moreover, there is a commitment to finding ways to integrate digital technologies into other course topics, expanding the horizons of archaeological survey and activity areas analysis.

Through this, the ripples of our experiment extended far beyond the Lakefill. Looking ahead, the future holds exciting prospects. Students will have the opportunity to deepen their analytical skills, explore cutting-edge equipment, and delve into the dynamic relationship between technology, data collection, and analysis. As we reflect on the lessons learned, the Media and Technology Innovation team gained valuable insights into student engagement and reaffirmed the potential of drone survey strategies in education and historic preservation.

This rewarding collaboration has allowed the Media and Technology Innovation team to demonstrate the usefulness of emerging technologies for the benefit of Northwestern students, staff, and faculty. As we soar into the future of virtual and immersive experience, we hope this lab experience will be a beacon of inspiration for other departments across the campus to embrace technology and provide dynamic, entertaining and educational experiences for our incredible learning communities.

This Spring, as the Innovator-In-Residence for Media & Technology Innovation (MTI), I curated our first virtual reality (VR) exhibition in the ENGAGE VR application. In pursuit of our goal of promoting the use of immersive digital storytelling technologies, NUIT and Media & Technology Innovation will be developing rotating virtual exhibitions to engage Northwestern University students, faculty and staff. We will be collaborating with members of the NU community to digitize some of their favorite objects and curate them within dynamic and interactive VR environments. To accomplish this, the project made use of digitization techniques including 3d photogrammetry and modeling, videography and digital curation to produce the inaugural MTI virtual exhibition in the ENGAGE VR metaverse.

Three members of the Northwestern University community, Dr. Amal Hassan Fadlalla, Zoran Ilić and Dr. Mary Weismantel, graciously lended their objects for digitization. Dr. Fadlalla, a University of Michigan Professor, contributed a Hadendowa Knife which she collected in Sudan during the late 1990’s. Fadlalla received her PhD in Anthropology at Northwestern University and donated the knife to the former Chair of the department, Tim Earle. I first encountered the object during an exploration of the Chair’s Office in 1810 Hinman. In addition to its stunning craftsmanship, the knife also encompassed a beautiful narrative of gratitude and exchange. The Anthropology Department stored the knife with a letter penned by Fadlalla in March 1999, which tells the story of the knife’s creation as a gift for Tim Earle and his assistance in helping Fadlalla obtain residency status to pursue graduate studies in the United States. The digital nature of the exhibition allows for audience members to approach the letter, read Fadlalla’s words and intimately experience the materiality of the 3d model.

The exhibition also includes a model of a 1952 edition Croatian Pinocchio Book. This object was donated by Zoran Ilić, Senior Academic Systems Engineer for NUIT and a vital contributor to the production of the exhibition. Ilić and myself conducted photogrammetry on each object in the Emerging Technologies Lab which allowed for the creation of 3d models that can be interacted with in VR. Ilić brought the Pinocchio book to the US from Croatia in the early 2000’s and I was able to interview him on the importance of the object within his family’s cultural heritage. He also took the time to explain the uniqueness of his collection and some of the ways in which the Croatian version of the classic story differs from the popularized Disney-fied edition.

The third object of the exhibition is a Peruvian ceramic Trujillo beer bottle collected by Dr. Mary Weismantel, professor of Anthropology at Northwestern University. This beautifully rich object was contextualized in a short interview with Weismantel, where she explains the symbolism, politics and anti-racism embedded within bottle’s creation. Weismantel is a preeminent scholar of the Andes – learn more about her work and the cultural context of the ceramic bottle in her book Cholas and Pishtacos: stories of race and sex in the Andes.

This project was a pleasure to create. As our first attempt, we were both inspired by the potential of these technologies and also frustrated by some of the difficulties that digitization and photogrammetry present. In the future MTI hopes to improve our processes of data capture and the resolution of our models, through the use of light diffusers, polarizers and tweaks within our studio lab setup. We are also looking forward to hosting the owners of lended objects within the VR application (from wherever they may be in the world) so that they themselves can share their stories and memories with a live VR audience.

I am extremely grateful for the time and patience of the object lenders, and the labor of NUIT staff such as Stephen Poon, Nate Bartlett and Zoran Ilić who assisted with the production of the exhibition. If you are a student, staff or faculty member at Northwestern University and would like to see your one of your favorite objects in a Virtual Reality exhibition, feel free to send an email to craigstevens@u.northwestern.edu

The Emerging Technologies Lab in collaboration with Kellogg School of Management had the opportunity to record a 360 video tour of a thriving and growing business as part of a classroom case study. We were able to record inside a Patel Brothers grocery store and create a VR tour throughout the space, showcasing its variety of produce, spices, and other specialty foods.

We hope this can lead to further classroom experimentation and exploration for immersive learning! 

As an art historian, this pairing of technological innovation and crisis of public morality was familiar to me. It happened with the printing press in the 15^th century, and again with photography in the 19^th. Who would buy a portrait when you could capture the exact likeness of a person with the technology of a camera? With generative AI, however, this historical precedent was complicated by the technology’s own inherent ability to produce original works with minimal human intervention. Whereas a camera requires lighting, focus, staging, and compositional direction from the artist in order to produce a photo, all Dall-E requires is a sentence to produce wildly variable images. What does this expanded field of creative technologies mean for arts practice of the future?

Enter Helicon, Northwestern University’s on-campus literary magazine. Established nearly 45 years ago by three dedicated poetry students, Helicon is a storied institution of literary and artistic merit representing some of the University’s best creatives. I reached out to the magazine’s editor, Lily Glaubinger, with a proposal: bring the editorial board to the Emerging Technologies Lab for a writing workshop, and see what original works can come from experimenting with this newly accessible technology. How would our writers use Dall-E2? What challenges would they face? How would generative AI influence their writing practice? Our contributors— Ann Gaither, Lily Glaubinger, Natalie Jarrett, Skye Tarshis, and Alivia Wynn— were eager to explore the implications of AI image generation for their own creative practices.

Lily Glaubinger

On a Saturday afternoon in January, Helicon’s editorial board arrived at the Emerging Technologies Lab to use Dall-E2 for the first time. We began with a short discussion of the state of generative AI in the field, a demo of how Dall-E2’s interface worked, and tips on how to tailor a prompt for the best results. We also discussed some of the major pitfalls and ethical concerns of generative AI, ranging from its tendency to depict femme-coded bodies in overtly sexualized ways, to its statistical likelihood of defaulting to white and lighter skin tones. By discussing such obstacles at the beginning of the writing workshop, we flagged such context as critical to our writers’ understanding of the tool at hand. Though generative AI can produce endless variations of images, text, sound, and even video at the simple request of a well-worded prompt, those outcomes are inflected by the biases and blind spots of the coders who build the structures that inform how a generative AI makes decisions. By knowing this from the start, we hoped to encourage our writers to think critically and adaptably about how they used the tool over the course of the workshop.

Ann Gaither

The resulting text-and-image pieces, displayed at the front of the Main Library in Spring Quarter 2023, are as beautiful as they are strange, uncanny as they are familiar, and humorous as they are disturbing. Each writer took a different approach to their use of Dall-E2; some specified clear descriptions with defined aesthetic parameters, while others fed their original prose into Dall-E2’s text box and let chance play out in ones and zeros. Some embraced the absurdity of the resulting images, others made hypotheses about how to produce certain types of images and experimented accordingly. In their exit interviews, almost every writer remarked on both the usefulness of the tool for the ideating process of creative endeavors, and their surprise at the limits of Dall-E2’s scope. The future of generative AI in creative practice and industry continues to take shape; it is our imperative to make sure that shape forms responsibly and ethically.

Though the exhibition has since been deinstalled, the project has been digitally archived in a beautiful web iteration here.

This blog post is technical in nature, and goes in-depth on the efforts put into a recent project.

Image of the trophy from the camera used to do the photogrammetry

Behind the scenes photo of the trophy on vinyl background

Zoran and I tried to do a photogrammetry scan of this Coffee Maker/trophy. It is a highly reflective surface, and has many curves as well, so we knew this would be challenging. Zoran tried photographing it in the 32″x32″ light box, but the trophy was very large, and there were unflattering reflections resulting from the white and silver sides of the light box, but also from the hole in the front, as the trophy reflects the camera as well.

We then tried setting it up on the white vinyl on a table top, holding the vinyl up with a C stand arm. The white was nice for behind the trophy, giving it nice contrast, but because it was a small piece of vinyl, it created an unflattering reflection in the bottom of the cup, similar to the reflection above. For lighting, we used two Litepanels Gemini 2×1, one with a softbox, and one without, to see if there was any significant difference. We couldn’t detect one.

The black vinyl had a similar problem, so we tried putting the trophy directly on the floor, using a 6×6 black flag that we hoped would be less reflective than the vinyl. We hung the vinyl behind the trophy.

This kind of worked, but the text on the trophy was really hard to read, so we put a white card on the floor, and that helped make the text more readable, but the white card was too small, only making some of the text more readable, despite being about a 3’x5′ board.

Wider behind the scenes shot of the trophy, showing lighting, camera and backdrops.

To scan a one-of-a-kind artifact like this, as-is, requires a complicated setup.

• One option would be to construct a larger white lightbox, and light it from the outside through diffused white fabric, and obscuring the lens when it peers through the fabric.
  • Something that could help would be a two way mirror in front of the lens to block the reflection of the lens, though this might result in seeing the trophy instead.
  • We could also be more creative in our lighting, perhaps lighting it directly from above, and using the tent/lightbox to illuminate it from the sides by reflection off the fabric.
  • One thing to consider is that the smaller the diameter of the “tube”, the smaller the reflection, but at some point the reflection is less of an object, and more just a specular highlight. It may be difficult to get an even lighting over the whole object because of this.
  • We may be able to build a larger lightbox with our newly acquired 6×6 frames and rags, but we have not tried. We could suspend the light source over the lightbox using the Menace arm kit. This would take up a lot more space in the studio than the current lightbox does, however.
• Another option would be to get the top parts of the trophy looking good, perhaps using a polarizing filter on either the camera, lights, or both. [see the sketchfab article below] Then photograph it from all angles, then light it so the middle looks good, then photograph it from all angles, and then light it so the bottom looks good, then photograph it from all angles. Then the three images for * each* camera and turntable position would need to be combined into one image, and then those composite images sent to the photogrammetry software for processing. This would require very precise camera and lighting positioning.
• If we would be able to alter the artifact, we could spray it with one of these sprays:
  • https://www.capture3d.com/3d-metrology-solutions/consumables/aesub-blue-vanishing-3d-scanning-spray
  • https://www.conrad.com/p/3d-coating-spray-david-coating-spray-500-1218840
  • These sprays change the shiny surface to matte, which means that the images can be captured much more easily, as the reflections are diffused. The sprays either need to be cleaned off, or they evaporate/disappear after a certain amount of time. It is not known now if these sprays would damage the trophy, Zoran is looking into that.
  • Then after the photogrammetry was complete, the surface texture would need to be rebuilt in Blender, but the overall shape would be correct.

The MTI team produced this video in the early weeks of sheltering-in-place to aid our colleagues, particularly faculty who are creating course lectures, in turning their at-home environment into a set for recording video content:

Since this video was produced, we’ve noticed there are still common pitfalls many recordings fall into. Fortunately, they have easy solutions. Here is a non-exhaustive list of issues you can look out for when recording yourself.

The Camera Is Too Low

Raising your computer’s camera to the same height as your eyes will produce a better image for presenting yourself to an audience. Not sure if it feels like the correct height? One easy tell is if you can see the ceiling in your image. See the ceiling? Your camera is still too low! Add another book/box or two to raise it up.

Poor Eye Line

Realistically, it’s difficult to maintain constant eye contact with your camera. In some cases, the camera (especially on laptops) is so small it feels like it’s not even there. To make matters more challenging, you can’t be expected to memorize or perfectly execute every lecture you need to record. It can help to position your outline or script near the top of your screen, as close as possible to the camera. Place yourself further back from your screen to help minimize eye movement. You can also record your video in separate clips to focus on individual chunks and use slides or other graphics on screen to strategically hide edits.

Poor Audio Quality

Are you recording audio for a powerpoint presentation, or your options for recording are limited to a platform that significantly compresses audio? You can vastly improve the audio quality of your recording by using your phone to record yourself!

Missing/Undesired Results for Video, Presentation, or Audio Content

Especially if you’re recording for the first time, do a brief 5 second test with your setup. Play it back to make sure everything you want to capture looks and sounds as you prefer. A brief test will go a LONG way to prevent you from re-recording an entire lecture.

In addition to these tips, remember that everyday occurrences such as outside noise, family, pets, and roommates have become a regular part of connecting with audiences virtually. Reduce distractions where and when you can, but in some circumstances there may be instances outside of your control.

COVID-19 has really impacted the way we work and think about working from home. We hope this helps you with your own recording, teaching, and learning.

Northwestern IT was on site to live stream the evening’s event in collaboration with Global Marketing and the Weinberg College of Arts and Sciences.

Last Friday we were delighted to join about 2000 wildcats to create this year’s class photo with a DJI Mavic 2 Pro drone!

Here are some of the behind the scenes photos.

Go ‘Cats!

We also filmed our own aerial footage of drones that Assistant Professor Michael Rubenstein has built with students. He will be teaching a new drone class this fall. Check back for a final video in October!