Léa Dorne still vividly remembers visiting a Syrian refugee camp and standing inside a family’s United Nations trailer in Jordan.
But she wasn’t actually in the Middle East. Dorne observed the family’s reality through an Oculus Quest headset while watching a film shot in virtual reality (VR).
“It’s very different than seeing a video clip on your smartphone,” said Dorne, a senior majoring in music business at the University of Miami. “When a movie surrounds you [in virtual reality], you can’t escape it. Therefore, the impact is multiplied. It’s much stronger, bigger, and more emotional, and these films can get a message across in a more impactful way.”
Dorne saw the film, “Clouds over Sidra,” as part of an Interactive Media class in the School of Communication. The experience solidified her decision to apply for and accept a Plus One Scholarship to spend an extra year at the University learning how she can blend immersive technology and music to amplify the power of film.
“Virtual reality has had the impact of making me think about other people and going to places I couldn’t go,” pointed out Dorne, who is also developing a virtual reality application with the programming help of Varun Krishnan, first-year Miller School of Medicine student. “In terms of education, that’s so powerful.”
She is part of a growing community of University students and faculty and staff members exploring the use of immersive technology—which includes virtual, augmented, and mixed reality—to improve the world around them. The University’s XR Initiative, launched in 2018 with Plantation-based Magic Leap, continues to evolve across three campuses as more students and faculty members develop projects that utilize immersive technology and can be viewed on a growing variety of devices and headsets.
“The Internet will become immersive, and this is how we will experience things online,” said Kim Grinfeder, director of the XR Initiative and chair of the Department of Interactive Media.
It comes at a time when immersive technology is becoming a critical tool to engage in the metaverse—or a virtual version of society—which could be even more convenient for students, professionals, and educators in times of necessary isolation.
Learn in a virtual location
For example, students can attend classes and learn in a virtual location of their choice using virtual reality headsets. This is something that Grinfeder, along with William Green, professor of religious studies; and Denis Hector, associate professor of architecture, are exploring this semester in the second edition of their course, “Religion and Sacred Space in the Era of Artificial Intelligence and Virtual Reality.” They hope that every class can be held in a different sacred space around the globe, with both students’ and professors’ avatars sitting around a virtual discussion table (using University-owned Oculus Quest headsets to attend).
Using augmented reality (or AR) applications, digital images or information are layered on top of our physical environment. These apps can be viewed through either a phone or tablet’s camera or by using headsets like the Magic Leap One or the Microsoft HoloLens 2. Today, AR is used widely to create architectural models that allow users to touch or click on certain areas of a building and learn more about them, said Rodolphe el-Khoury, dean of the School of Architecture.
Last fall, students in the senior design studio class displayed AR models of their proposal for Cutler Bay’s new Town Hall and a neighborhood redevelopment project near Atlanta at the School of Architecture’s Homecoming tent. Using tablets or their own phone cameras, visitors could see an augmented version of the students’ design, with information like the water and elevation conditions for each site and expected population density. They also could visualize greenspaces within the model.
“We are investing in emerging technology in a variety of platforms to ensure greater access to immersive learning experiences,” said el-Khoury, adding that the school is also utilizing virtual reality and that lecturer Ruth Ron’s Introduction to VR and AR class has gained a steady following. “At Homecoming, we wanted the students’ designs to be as accessible as possible, so that people could use their phones as a lens through which they could view and interact with the project. When headsets become more pervasive, this will be an even more immersive experience. And we are preparing students for that reality, which we anticipate will be prevalent in the future.”
Grinfeder has watched the desire to utilize immersive technology steadily progress since he worked on his first VR project in 2014 at the School of Communication.
“The University of Miami is well-prepared and ready when it comes to immersive technologies,” he said. “And I’m thankful that the administration had the foresight to notice how important these technologies are and to invest in them. That was very forward-thinking, and it’s why we’re not behind the curve today.”
Grants offered to build new XR products
Each year, through the XR Initiative, more students and faculty and staff members who want to explore the field are learning how to incorporate augmented reality, virtual reality, and mixed reality into their research and scholarship. Those with an idea to advance their field of study can apply for funding through the University’s XR grants and the Provost XR Awards.
So far, 32 faculty members and 11 students have received funding to produce XR applications. Dorne received a grant to build an app inspired by synesthesia: a neurological condition where two of a person’s senses are experienced simultaneously—like if certain sounds elicit images or colors. In her app, users can play different notes on a drum, and each note elicits a different colored firework on the horizon. Soon, the drum notes are powering a ship forward through the ocean, and the player engages in a rhythm duel with an enemy ship.
“My intent was to incite a spark in people’s creative minds to establish connections between two forms of art,” said Dorne. “So, the user can be playing a drum set, and by playing certain patterns or sequences it triggers a visual narrative.”
Berk Basarer, a senior studying electrical engineering, is launching “AR Translate,” an application that offers users real-time language translation using an augmented reality headset, so that people could communicate even if they do not speak the same language. With the app, users would see real-life subtitles of what the other person is saying and could express themselves in their preferred language.
Gaining firsthand knowledge
Currently, at least 26 University courses incorporate virtual and augmented reality into their curriculum. There are also several places on campus where students and faculty members can experience immersive technology.
All University students and faculty and staff members have access to the XR Community Lab on the third floor of Richter Library, although appointments are recommended. In addition, students and faculty members in the School of Communication can use the XR Studio, and students and faculty members at the School of Architecture can visit the Responsive Architecture and Design (RAD) Lab and its VR/AR facility.
On-the-ground training in how to program for virtual reality (typically using software called Unity) can be gained through classes at the School of Architecture, School of Communication, and in the College of Arts and Sciences Department of Computer Science.
Students who want an accelerated experience can also serve as an apprentice in the UMIT Innovate program. Director Max Cacchione leads a team of student programmers at UMIT Innovate who work with students and faculty members to bring their extended reality ideas to life. Although some of his first trainees were computer science and software engineering students, he said, interest in learning how to program for virtual and augmented reality expanded to other corners of the University.
“There’s more student interest in learning how to navigate immersive technologies, and each year our team grows about 30 percent,” Cacchione said. “Students benefit from the program by obtaining a job at UM, a Unity Certification, and experience working on an agile software team. The University also benefits from the creation of intellectual property and grant opportunities through projects driven by faculty who employ UM Innovate students.”
Computer science professors Ubbo Visser and Viktor Milenkovic agree. Visser teaches an introduction to game programming course that is open to all undergraduates and is always full. Milenkovic teaches classes in computational geometry, or the mathematics behind 3D programming for virtual and augmented reality. But he also serves on the XR Initiative’s advisory board.
“We teach the principles, so people understand what’s going on and what’s possible in this medium,” said Milenkovic, who is also chair of the Department of Computer Science.
Some of their particularly skilled students often do an independent study, and two graduate students recently worked with Milenkovic and Visser to finish an XR application. Their app, “Intuitive Interface for Human-Robot Interaction,” uses augmented reality headsets to create a personalized 3D map of a house, which is then used to train human support robots to navigate an elderly person’s home.
“Someone will come in wearing AR goggles, and they will map out the home and furniture. So, the robot knows where their bed is and where the bathroom is, for example,” Milenkovic explained.
A world of opportunity
As XR equipment and technology become more accessible, faculty members are coming up with valuable ideas about how to integrate the technology, according to Cacchione, which is keeping his student team very busy.
The UMIT Innovate team is now starting work on an application—devised by engineering lecturer Diana Arboleda and associate professor James Giancaspro—to teach statics. This is an area of physics critical to young engineers that is difficult to comprehend in a 2D textbook but is much clearer in 3D, which is possible using augmented reality. This collaboration was recently awarded a $300,000 grant from the National Science Foundation.
Faculty members can also enlist the help of professionals at the Institute for Data Science and Computing (IDSC) to develop applications in extended reality. Manouj Govindaraju, a software engineer at IDSC, is working with several faculty members who received XR grants this semester. “Smart Assistant,” conceptualized by communications professor Ching-Hua Chuan, will help line workers train and refine their skills using artificial intelligence and augmented reality headsets that will track each step of a process and then redirect the worker if they forget a step, helping them to become more efficient. Another application devised by Delia Cabrera DeBuc, research associate professor of ophthalmology, and Dr. Ranya Habash, assistant professor of clinical ophthalmology, aims to help clinicians diagnose neurodegenerative diseases, like Alzheimer’s, at an early stage. The app will simulate a virtual reality grocery store trip and then track a patient’s brain activity to see if they struggle to remember what they need to buy.
Grinfeder is pleased to see a rise in student and faculty enthusiasm for immersive technology, which grows as more useful ideas are proposed each semester.
“That Magic Leap launch really pushed everybody at the University to think about immersive media and what the metaverse could become, and now we’re starting to see the fruits of it,” he said. “A lot of the applications we’re currently getting are well-grounded ideas that demonstrate an understanding of how these technologies can be leveraged. And while this idea of the metaverse might have caught some people at other institutions off guard, here at the University of Miami, that’s not the case.”
