Lassonde alumnus wins CIPPRS John Barron Doctoral Dissertation and Info-Can Canadian Computer Science Distinguished Dissertation Awards

Dr. Mahmoud Afifi, a recent graduate from the Department of Electrical Engineering & Computer Science, has won two prestigious awards for his PhD dissertation.  First, he received the CS-Can/Info-Can 2021 Canadian Computer Science Distinguished Dissertation Award. This award recognizes excellence in research and writing by a computer science doctoral student completing their degree at a Canadian institution. Next, he received the 2021 Canadian Image Processing and Pattern Recognition Society (CIPPRS) John Barron Doctoral Dissertation Award, given annually to the top PhD thesis in areas covered by the Conference on Computer and Robot Vision (CRV) including computer vision, robot vision, robotics, medical imaging, image processing or pattern recognition. 

Supervised by Professor and Canada Research Chair, Michael S. Brown, his winning thesis was titled, ‘Image Colour Correction, Enhancement and Editing.’  “I’m happy to see Dr. Afifi’s hard work, creativity and scientific efforts have been acknowledged by these two awards,” said Professor Brown. “It was my honour to be his supervisor.”

When using a smartphone camera or a professional DSLR to take a photograph, an image is captured when the shutter opens and light falls on the digital sensor. The photograph begins its journey as an array of raw electrical signals and then undergoes a chain of processes applied by an image signal processor (ISP) to produce the final output. Typical processes include noise reduction, contrast enhancement, image sharpening and colour correction. These types of processes are termed low-level image processing, in contrast to high-level image processing, which focuses on classification or providing a description of what is in the picture. Afifi’s work was focused on understanding and implementing colour correction processes to produce colour-accurate images from digital cameras.

One part of his thesis was to correct the colour of images with inaccurate white balance, that is, how warm or cool the overall colours of a photograph look, especially the parts that are, or should be, white. If you’ve ever tried to take a photo of a scene lit simultaneously by office fluorescent lights and sunset streaming in from a window, only to discover that your image has an unpleasant bluish or reddish cast, then you have encountered a white balance problem.

“Whether it’s because of the white balance tints or the effect of lighting on the image, our goal was to try to make white objects look white,” explains Afifi, a task that is more complicated than it may seem. He proposed several methods for white balance correction and editing. He worked with Adobe Research to correct images post-capture after the photograph has already been taken. Another area he worked on with Adobe was auto recolouring a photo based on the objects of the scene, a significant evolution from typical recolouring filters, like those found in Instagram, which do not understand the contents of the image.

The second aspect of his research was colour correction for exposure errors. A photographer usually adjusts several camera settings to make sure that the right amount of light is let into the sensor (the exposure) to accurately capture the colours of an image. Underexpose the image, and navy blue or dark brown objects start to become indistinguishable from dark, black shadows. Overexpose the image and pastels get washed out and appear white. Even when you try to correct the exposure using post-production photo editing tools, the software may not be able to restore the original colour of the object as that information has been lost.

Afifi tackled this problem of colour degradation arising from exposure errors, carrying out an internship with Samsung to work on the post-capture correction.

Afifi also worked on illuminant estimation during an internship with Google, where he, along with researchers from Google, designed a neural network to perform illumination estimation, and moreover, developed a sensor-independent approach so that the model does not need to be rebuilt or retrained for different sensors or camera hardware.

The consequence of accurate colour and white balance corrections not only help with producing better-looking images, but they also provide better inputs for downstream processes such as object identification within photographs. Although deep learning approaches that are used in such high-level image processing are already fairly accurate, they are very sensitive to the reference data used to train the neural network model. “If the training data used to identify cats are all photos taken with good white balancing, the deep learning model may not be able to accurately identify cats in images with inaccurate white balance,” explains Afifi. He addressed the problem by developing a colour augmentation technique to improve network robustness against images with white balance mistakes.

Afifi’s research has resulted in thirteen papers, five patent applications, a collaboration with Adobe, and internships at Samsung and Google. Afifi credits his success to his experience at Professor Brown’s lab, where he was able to learn to treat problems from different angles in a supportive research environment. “If you have a deep understanding of the problem that you like to solve, you will find it easy to get innovative and think outside of the box,” says Afifi.

While he found research to be fun, at times it was also challenging. “You have to keep the passion up, deadline to deadline, idea to idea and with short timeframes,” says Afifi, “and it was also challenging to find that balance between the research and my kids, wife and family.”

When he applied for these awards, Afifi did not think he would get them but was ecstatic to learn the news from Professor Brown. “Even my daughter told her friends,” says Afifi. “This achievement is definitely a happy ending for this journey.” Mahmoud Afifi now works at Apple in colour processing as a Camera Algorithms Engineer.