Research from Lassonde Professor explains mechanism that allows us to see what we are not looking at

Covert shifts of attention help us perceive day-to-day activities where visual events happen quickly, such as driving, riding a bike or playing a video game, but do not permit enough time for eye movements to focus on every detail. This process allows us to change our visual attention without shifting where our eyes are looking. For more than four decades, Professor John K. Tsotsos from the Department of Electrical Engineering and Computer Science at York University’s Lassonde School of Engineering, has focused his research on computational and human vision, specifically visual attention. Part of this research involves the development of a theory to predict exactly how our visual cortex – the portion of our brain that processes visual information – performs the complex functions that make covert shifts of attention possible. Tsotsos has also developed computational algorithms and robots to demonstrate his proposed theory. Recently, neuroscientists reported new evidence for the exact mechanism that was proposed by Tsotsos. 

To achieve covert shifts of attention, our visual cortex must determine the exact locations in our visual surroundings that are of interest. “Imagine a vanilla glazed donut placed in front of you, and you keep your eyes focused on the center of the donut hole,” says Professor Tsotsos. “While your eyes are fixed on the center, the donut is replaced with a similar donut, except this one is decorated with sprinkles. The sprinkles are arranged in eight lines, all pointing in different directions and equally spaced across the top of the donut. Without shifting the focus of your eyes from the center of the donut, we can covertly shift our attention to identify the location of the sprinkle that is pointing vertically.” The computational mechanism used to achieve this example of a covert shift of attention was proposed in Professor Tsotsos’ Selective Tuning theory of visual attention in 1995. 

In a long-standing collaboration with neuroscientists in Germany, pivotal research that tested this idea was recently concluded, supporting Tsotsos’ proposed mechanism. This research was led by Dr. Mandy Bartsch and under the direction of Prof. Dr. Jens-Max Hopf of the Leibniz-Institute for Neurobiology and the Otto-von-Guericke University, in Magdeburg, Germany. This work revealed that covert shifts of attention are achieved by a progression of focus from high to low levels in our visual cortex hierarchy – just as Tsotsos predicted, almost 30 years ago. 

This conclusion was reached by investigating activity in the visual cortex during covert shifts of attention, using a series of visual search tasks. Participants were asked to keep their eyes fixed on an image in front of them, while distinguishing colour-defined targets among other items appearing in nearby quadrants. During these tasks, a neuroimaging technique called magnetoencephalography (MEG) was used to assess activity in the visual cortex, specifically focusing on a signal that MEG detects, called N2pc. This signal and its detection permit the exploration of the underlying mechanism of visual-spatial attention.  

Experimental design of visual search tasks, with image of the cross where participants were told to keep their eyes fixed, and colour-defined targets in surrounding quadrants that were covertly distinguished according to experimental conditions.

When participants covertly shifted their attention to detect a target, initial activity was observed in higher levels of the visual cortex, eventually progressing to mid and lower levels. Even when participants shifted their highly concentrated focus from one target to another, activity in the visual cortex would restart from high to low – demonstrating that attentional focus does not simply shift, it restarts a computation beginning in higher levels in our visual cortex. This investigation showed that covert shifts of attention can be attributed to a process that is much like zooming-in on a camera, where the lens begins with a broad target, followed by a more selective focus on finer details in the frame – also known as coarse-to-fine process.  

Beyond the contribution of important knowledge to the field of cognitive neuroscience, these research findings can be used to enhance design approaches for advertisements and visual user interfaces for electronic devices. By incorporating knowledge of how human viewers shift their attention covertly, designers can improve the spatial arrangement of important information to maximize what viewers see. In addition, this research may lead to more accurate and efficient machine vision systems that behave more like human vision. 

The manuscript was published on March 8, 2023, in the journal Science Advances and can be viewed online: A cortical zoom-in operation underlies covert shifts of visual spatial attention.