What do people look at when they view a painting?

When you look at a particular scene or painting, what areas or objects do you fixate on? Does everyone look at a painting in a different way, or is there some generic pattern or strategy that most of us follow? These questions have kept cognitive psychologists or perception scientists busy many times before. Most earlier studies into this topic presented images of paintings to participants on a computer screen, while their gaze was tracked with stationary and often bulky eye trackers. These require participants to sit down and keep their head extremely still, because the slightest head movement can throw off the tracker's calibration. And of course, these types of studies need to take place in a lab, because the eye trackers are not easy to carry around.

This is not really what one would call a natural setting to study viewing behavior in. Normally we have the ability to walk around, and often we also turn our heads in the direction of the things we want to look at. Luckily, light-weight mobile eye trackers have started to hit the market in the last few years and they allow researchers to study participants in less restricted settings. Most of these mobile eye trackers are really expensive (think of $50,000+), but lately some vendors have started to offer options that are quite affordable. One of these is the Pupil mobile eyetracker.

Its development started out as a project for a Master's thesis at MIT to see if it was feasible to create a cheap mobile eye tracker with components extracted from webcams, but it eventually led to the establishment of the Berlin-based company Pupil Labs. The Pupil eye tracker can be purchased for between $1,500 and $2,500 and only weighs around 50 grams. It plugs into a laptop with two usb cables (one for the eye camera and one for the camera filming in the direction the wearer is looking at; the more expensive models have both integrated into one usb-connector). The recording software is open-source, well maintained and free to download. The only thing that is currently missing in my opinion, is an officially supported software package that helps with quantitative analysis of the recorded data, but probably that is on the company's roadmap somewhere (In the meantime, I wrote a small library of my own for this purpose, but haven't maintained it really well). Pupil does however supply software that allows one to visually inspect the recorded data, Pupil Player, which is great (it borders on sci-fi).

With the pupil eyetracker in our possession at the Cognitive Psychology department of the Vrije Universiteit of Amsterdam, Francesco Walker came up with a nice idea for a research project. Besides his interest in human perception, he also likes art, so why not combine the two? His suggestion was to use the Pupil to study how people appreciate visual art, or more specifically, look at paintings. He contacted various museums in Amsterdam to propose this project, but only the Van Gogh Museum (seriously, if you ever visit Amsterdam, don't skip it!) responded and was enthusiastic about it.

With our team consisting of Francesco, Berno Bucker, Nicola Anderson, Prof. Jan Theeuwes and me, we came up with the idea to see if and how background knowledge about a painting changes the way it is viewed. We first let participants free-view each of 5 different van Gogh paintings for 30 seconds, and after this first phase, we told them more about the background of each painting. We then requested participants to view the same five paintings again for the same amount of time; all the time wearing the Pupil eye tracker that registered their gaze. We were also interested if viewing behavior changed with age, so we recruited two groups of participants: One consisted of young adults recruited at the university and the other of children that were finishing elementary school.

Now you might think "of course having knowledge about a painting will change the way someone looks at it" or simply that people will look differently at a painting because they see it for a second time, and you are probably right. However, we were particularly interested if a general pattern in viewing behavior, shared by most participants, would emerge during the second round of viewing. For instance, if participants knew that Van Gogh made a particular painting during one of his depressions, would they tend to pay more attention to areas or objects in the painting that are indirectly indicative of his mood, such as cloudy or dark skies?

The study was quite challenging to perform, especially since we had to conduct it during the normal opening hours of the museum, while a lot of museum visitors were walking around that were very intrigued by what we were doing. Luckily this didn't negatively impact data collection and we had some interesting findings. In the first phase, the viewing behavior of the children was mostly driven by the properties of the scene, while the adults showed more goal-driven viewing behavior. That is, the adults tried to view the paintings with a purpose, while the children just looked at areas in the painting that were more salient or conspicuous. In the second phase, the viewing behaviors converged; both groups looked at the paintings in a more goal-driven fashion. For some these results may be a bit unsurprising, but I think it is quite cool that we have been able to conduct a study with which we collected reliable quantitive data outside of a laboratory and in minimally controlled settings. There is still room for improvement, for instance by eliminating the necessity of the QR-like markers that designate the locations of the paintings in the video feed, but hopefully what will be possible in the near future.

Here is a demo video of the study that also shows the abilities of the Pupil Player quite nicely:

You can find more information about this project in the article that has been published in PLoS One