Wrinkles, the position of the eyes, eyebrows and birthmarks. These are all characteristics that experts at the Netherlands Forensic Institute (NFI) in the field of facial comparison look at to compare individuals in photographs. These are often security camera images captured by suspects of burglaries, fights or robberies. If there is doubt in the criminal case about whether the person in the photo is the same person as the suspect, the expertise of the NFI is called in. Recently, in addition to experienced experts making visual judgments, the NFI also deployed a new method: software that supports the expert's judgment.
Is the unknown person in the disputed photo from a security camera the suspect who came into view? At the hearing, the opinions of the Openbaar Ministerie (OM) and the legal profession may differ on this issue. The judiciary can then ask NFI experts to investigate how likely it is that the person in the photo is the suspect.
Facial comparisons in photographs appear in all sorts of cases. From robberies to debit card fraud. And from brawls to individuals emerging as suspects in a war zone. The key question: is the person in these images the suspect? Arnout Ruifrok, with his nearly 20 years of experience, has a well-trained pair of eyes. "I look at the face shape, forehead, wrinkles, eye shape, corner of the eye, nose, nostrils, birthmarks and mouth. We go over the whole face: do we see similarities? Do we see differences?" The experts give a little more weight to distinctive details such as pigment spots, wrinkles and birthmarks. "We also look at the shape of the skull and the hairline," Ruifrok says. "The experts go down a list of characteristics."
The experts note from that list what they can see, what the similarities are and the differences. If features don't match, they describe what the reason might be. Is it because of the angle at which the photo was taken? Could it look different because of light or out of focus images? "We keep all those variables in mind," he explains.
To strengthen the experts' judgment, Ruifrok has been working on a new, objective method in recent years: software to compare faces. "Suppose you have two photographs in a case, the disputed image of a robber and the reference image of the suspect. I look at those as an expert, but now a software program is also 'looking' along. Like humans, the computer also 'looks' at facial features." A smart algorithm has been trained to do this because it has seen lots of images of faces in all sorts of poses and exposures. And like humans, the software is able to distinguish features, but more objectively. The software gives a score of to what extent the pictures are similar, expressed as a number. "That is a score between 0 and 1. 0 is totally different, 1 is exactly the same photo," Ruifrok informs.
Then translating that score into a probative value is more complicated. Ruifrok found that if you offer the software two poor-quality photos, according to the software's score, they are very similar. That's because the model sees a similarity: namely, both photos are of similar, poor quality. "You have to think for example of a blur, poor exposure or an odd angle. So if two photos are bad in a similar way, according to the software they are similar and it gives a high score," Ruifrok said.
Ruifrok had to do something with that. He developed a method that determines how much influence the quality of the photo has on the final score. If you know the quality of the photo, you know how much of the score comes from the similarities or differences in quality. After all, two images of poor quality also give a high score, because the model sees a high similarity because both photos have a blur, for example. But you specifically want to know if the score is high because the faces match.
So how do you find out? Ruifrok held the poor-quality photos, as we know from surveillance footage, against a database containing photos of various kinds of quality. The database contains only photos of faces of people who have given explicit permission to use them for the intended purpose. He compares his "bad" photos to this database to see if the quality of his photo is high or low. Again, the score applied was between 0 and 1. The lower the score, the fewer similarities between the two photos. Therefore, if he compares the disputed, poor-quality photo to the database and the score is low, it means it is a good-quality photo. After all, the system can distinguish the photo from all kinds of quality photos of other people. Is the score high? Then the software can no longer distinguish the photo from all kinds of other low-quality photos. This is because it is wrongly mistaken for the face of other persons because of the circumstances; the poor quality of the photos.
"When I know what the quality is, I test it on good quality photos like we know from passports or driver's licenses. That way I know what the photos score on average with respect to good quality photos," he explains. "Then I can calculate what the probative value is by comparing the probability of finding a score if it's the same person to the probability of the score if it's a picture of a different person."
The study shows that there is a good agreement between human judgment and software judgment. Both results will appear in the expert report from now on. Expert judgment will always coexist with that of the software, Ruifrok emphasizes. "As a human, you can explain and discuss differences. I see a wrinkle, did you see it too or is it shadow after all? And do you count it or not? As experts, we can discuss this with each other. You can't do that with the computer, it can't interpret the gray areas."
The new method allows experts to better assess the probative value of non-optimal photographs from now on. They help experts make a probability judgment about whether the results of the examination better fit the hypothesis that the person in the photos is the same or not. "I have been an expert on facial comparisons for 20 years. Yet I remain aware of the limitations I have as a human being. Like bias, bias. You have to acknowledge those, that's just how our brain works. That's why we always look at pictures of faces with three pairs of eyes independently. The fact that we are now also using the comparison software as a more objective method is a nice addition to that."
