What if audiences could not only hear flutist Valerie Coleman play her dramatic piece about the voyages of enslaved Africans bound for the New World, but actually float in the sea, amid the anguished faces of those thrown overboard?
Or if neurosurgeon Tibur Urakov could explain how he would repair a fractured spine by converting the grey blobs on a CT scan into a life-sized, three-dimensional animation that highlights the vertebrae, discs, and nerves in different colors right before his patient’s eyes?
These are not far-fetched fantasies but just two of the 24 diverse proposals that were awarded University of Miami grants to help faculty and staff members lay the foundation for the “Magicverse,” the new digital world of spatial computing that Magic Leap founder and alumnus Rony Abovitz envisions for his company’s mixed reality technology, which moves 2D computing into the 3D world.
When wearing Magic Leap’s high-tech goggles, users could tour a coral reef without getting wet; learn how to evacuate patients during a smoke-filled operating room fire without any real danger; or teach frustrated parents how to deal with their child’s temper tantrums—as three of the other funded projects propose.
In all, 71 teams comprised of a total of 154 collaborators from every school and college—as well as from UM Libraries, the Center for Computational Science, IT Academic Technologies, and the Department of Continuing and International Education—answered Provost Jeffrey Duerk’s call last fall for proposals that considered how mixed reality technology could impact their teaching, learning, research, scholarship, and artistic endeavors.
Jean-Pierre Bardet, vice provost for strategic projects, said he and Duerk were so impressed with the sweep and scope of the proposals that they agreed to support 15 projects—50 percent more than originally intended—and secured funding for an additional nine projects from the Sylvester Comprehensive Cancer Center, Clinical and Translational Science Institute, and College of Arts and Sciences. The selected projects were each awarded $10,000 in seed money or the help of a graduate student.
Almost half came from the Miller School of Medicine, which given its size and the many ways that augmented (AR) and mixed reality (MR) already are facilitating interventions and enhancing both teaching and the patient experience, wasn’t unexpected. But Bardet said the breadth of the proposals that could transform everything from how audiences interact with public art and museum exhibitions to the way engineering students learn one of the most fundamental, but difficult, engineering principles was impressive.
So, too, was the extent to which the proposals relied on multidisciplinary collaborations across schools and colleges, a sign that the burgeoning culture of interdisciplinary inquiry envisioned by the Roadmap to Our New Century and encouraged by the University’s Laboratory for Integrative Knowledge (U-LINK) is taking root.
“The proposals were really outstanding, in terms of innovation and in terms of the significance for the future of the University,” Bardet said. “We’re really proud of the explosion of ideas and the level of interactions across the schools and colleges, which we didn’t mandate. The principle investigators naturally sought expertise from other disciplines in other schools, so it’s quite possible that the seeds planted by U-LINK are blossoming into other initiatives.”
Jeffrey Marc Buchman, assistant professor and stage director for opera at the Frost School of Music, said his collaborations with the Center for Computational Science and the School of Communication are essential to overcoming the logistical challenges of his proposal for “Wish,” Coleman’s piece for flute and piano about the horrors of the Middle Passage.
Inspired by poet Fred D’Aguiar’s poem of the same title about the sea journey slave ships took from West Africa to the West Indies to sell their human cargo, Buchman knew as soon as he heard Coleman play the 11.5-minute piece last fall that it was meant to be viewed through Magic Leap’s headsets.
“It just hit me when she was about halfway through,” Buchman recalled. “I’m seeing her in this vast ocean floating with the trail of bones of the enslaved who were thrown or jumped overboard. I said, ‘Wow, what if audiences were immersed in that sea?’ It would raise the powerful impact of this powerful piece quite substantially.”
In one of his two medical-related Magic Leap proposals, neurosurgeon Urakov, an expert in using mixed reality in spine and brain surgeries, and his collaborators aim to help the hundreds of patients who undergo CT or MRI scans across the University of Miami Health System every day better understand their results. As Urakov noted, “To a medical professional these images are second nature, but to a regular person they are confusing blobs with 50 shades of grey.”
Urakov’s team—which includes Max Cacchione, director of innovation for information technology, fellow neurosurgeon Michael Ivan, who specializes in brain tumors, and students in the Departments of Software and Biomedical Engineering—is already working on a platform to enable Magic Leap devices to communicate with the health system’s Picture Archiving and Communication, or PAC, system to generate 3D mixed-reality projections of X-ray images on demand.
“Imagine how much happier patients will be when we can bridge the gap between the radiographic findings and their understanding of their problem by enabling them to view their brain, spine, and body in three-dimensional holographic representations,” Urakov said.
He and Ivan also are working on a second funded Magic Leap project that would enable neurosurgeons to incorporate 3D digital pathology and blood vessel flow dynamics in their current surgical navigation system and tools. The idea is not only to enable neurosurgeons to make intraoperative decisions more quickly, but improve simulation training for medical students and residents.
Other Miller School proposals address similar themes and purposes. They include a team of gastroenterologists who believe Magic Leap’s goggles could help them better perform and teach video-based endoscopies, and a team of interventional radiologists who propose using the headsets to map technically challenging maneuvers with catheters and other tools, reducing the need to administer successive doses of contrast agents and radiation.
Focusing more on the patient experience, orthopedic surgeon Lee Kaplan, the chief of UHealth Sports Medicine and head team physician for the Miami Hurricanes, is joining forces with the Center for Computational Science to create an immersive AR tour of his operating suites at The Lennar Foundation Medical Center. His goal is to “reduce anxiety and stress before surgery” by familiarizing patients with the sights and sounds they’ll hear through the pre- and post-operative processes.
In conjunction with Sylvester Comprehensive Cancer Center, a team of psychology faculty from the College of Arts and Sciences also hopes to improve the patient experience by evaluating whether breast cancer patients would have better outcomes if they engage in positive social and cognitive experiences via Magic Leap headsets during their chemotherapy infusions.
Several other funded proposals aim to enhance teaching and learning across and beyond the University. One team with expertise in applied behavior analysis, autism spectrum disorders (ASD), computer engineering, and interactive media proposed using Magic Leap’s mixed reality to help adults with ASD develop employment skills.
Two assistant professors, Greta Mitzova-Vladinov, the associate director of the School of Nursing and Health Studies’ nurse anesthesia program, and Magda Aldousany, in the School of Education and Human Development’s Department of Kinesiology and Sports Sciences, plan to use Magic Leap’s mixed reality to, respectively, help students develop a systematic approach to setting up an anesthesia suite and to better understand the mechanisms of different athletic sports injuries.
In two other art-related Magic Leap projects, lecturers in the Department of Civil, Architectural and Environmental Engineering and the Department of Cinema and Interactive Media are teaming up to create “Machines of Loving Grace,” an interactive, public art experience that uses real-time biometric data, and a team from UM Libraries and the Lowe Art Museum is creating an AR experience to allow museum visitors to interact with the upcoming exhibition commemorating the 40th anniversary of the Mariel boat lift.
Kim Grinfeder, the director of the School of Communication’s Interactive Media Program, said it’s too early to fully understand the implications of the Magicverse. But not too early to be excited about them.
“I feel the same energy around these new immersive platforms as we saw in the early internet days,” said Grinfeder, who is on three Magic Leap teams, including those working on the “Wish” and ASD projects. “This funding opportunity is a great chance for students and faculty to learn more about developing products for spatial computing.”
View the complete list of the funded Magic Leap proposals.
