Lassonde researcher aims to make VR a reality

Gene Cheung, an associate professor in the Lassonde School of Engineering, is working to advance augmented reality and virtual reality through signal processing tools for point clouds that are relevant to rendering everyday objects in 3D spaces.

This story originally appeared on YFile on Jan. 13, 2022

Augmented reality (AR) and virtual reality (VR) have the potential to help us see the world in a whole new way; it could revolutionize health care, education, urban planning and even tourism. An associate professor in the Lassonde School of Engineering, Gene Cheung, is advancing our understanding of AR and VR through 3D image processing.

Working with industry partners such as InterDigital and Google, Cheung is building signal processing tools for point clouds, which are datasets comprised of discrete points in 3D space that represent geometric shapes of everyday physical objects like chairs, tables and human beings. The work is critical in advancing the field of VR because truly immersive 3D imaging remains unrealized.

“We haven’t seen high-quality 3D imaging yet,” says Cheung. “But advances in point cloud processing is one of the main missing pieces, and if we can solve that challenge it will bring us much closer.”

He explains that TVs and movie screens are often marketed as 3D, but they are actually rendered in 2.5D. The way 2.5D works is by providing a stereoscopic view, where the left and right eyes observe slightly different viewpoints, giving an impression of 3D. However, the technology behind 2.5D imaging lacks motion parallax, a depth-perceptual cue where objects close to the observer move more in the observation view than the background as one shifts his or her head.

Graph signal processing, one of Cheung’s areas of expertise, offers a solution for processing 3D images. Modern cameras collect a large amount of rich data, including information about scene depth and the distance between the camera and observed objects. This data provides information that can be used to render an object from various viewpoints. However, data acquired from a camera also tends to be noisy and incomplete due to sparse and unreliable point sampling. Leveraging graph spectral theory, Cheung and his team are designing fast graph filtering algorithms to reduce noise, super-resolve and Inpaint (a graphics software for retouching photos and removing unwanted objects) point clouds.

The research, which began in 2010, is already demonstrating significant improvement over competing algorithms for restoring point clouds. While there are still ways to go in advancing AR and VR technology, Cheung envisions it will have a positive impact on society.

“The potential for 3D in AR and VR is significant,” explains Cheung. “You can imagine so many applications from remote education to health care. How different would remote education be for students during the pandemic if they were able to put on AR headsets and experience learning as if they were physically in a real classroom with their instructors and friends?”

The potential for health care is also important and Cheung’s research could be a game-changer for remote communities with little to no access to services.

“I have friends in Australia who are doctors and have to travel for hours to rural areas to treat patients because some communities don’t have access to specialists,” he says. “VR could provide more equitable access to services, and enable doctors to diagnose, and even perform surgery.”

He says his collaborations with industrial partners are important to bridge the gap between theory and practice.

“Quite often I work with people who are domain experts in a particular area and would never think of using graph signal processing theory. You need people with different areas of expertise to make good research happen,” he says.

Cheung, who considers himself an applied mathematician, finds inspiration in the beauty behind math and the mathematical derivation of theorems.

“In my experience, the best solution tends to be the simplest, most elegant one. The beauty in math is that you cannot really invent it – you uncover it as if it has always been there all along, waiting to be discovered,” he says.

Cheung joined York University in August 2018 and is a core member of Vision: Science to Applications (VISTA), York’s revolutionary vision science program. He became a Fellow of the Institute of Electrical and Electronics Engineers in January 2021. Prior to joining York, Cheung was a senior researcher in Hewlett-Packard Laboratories Japan and an associate professor at the National Institute for Informatics in Japan. He obtained his PhD from UC Berkeley in 2000.