Anti-spying software Proctorio works significantly less well with people of darker skin color. Thus, there is implicit discrimination, experts say. Proctorio says in a response that the research is "not an accurate representation" of how the system works. This is evident from RTL News' research into the operation of the exam software
Surveillance software designed to monitor students taking exams is also known as proctoring software. Such programs can take pictures and videos with the webcam, eavesdrop on the microphone, track what is seen on the screen, record keystrokes and view the student's browsing history. During the corona pandemic, many colleges and universities used this anti-spying software. Now that the pandemic is behind us, educational institutions are using this software for such things as college entrance exams, or students doing internships abroad.
In the summer of 2022, Proctorio, one of the best-known developers of proctoring software, made headlines. Robin Pocornie, a student at the Vrije Universiteit Amsterdam complained that the program did not recognize her as human because of her dark skin color. Fellow students with a white skin color, on the other hand, would not suffer anything.
Together with the Racism and Technology Center in Amsterdam, Pocornie filed a complaint with the VU and asked to stop using this software. When the Board of Governors did nothing with their complaint, they decided to go to the Human Rights Board. The latter will rule on the case on Tuesday, Oct. 17. Minister of Education, Culture and Science Robbert Dijkgraaf wrote in a letter to the House of Representatives that there is no evidence that Proctorio's anti-speech software discriminates.
Proctorio has commissioned an independent audit of its surveillance software. This would show that the face detection software does not perform worse on people with darker skin tones. However, the company does not want to reveal the report to RTL News. Therefore, the medium decided to conduct its own investigation into the program.
The conclusion does not lie, according to RTL News: Proctorio face detection works significantly less well on people with black faces. RTL News presented the findings to several experts. They support the outcome of the study.
"This research confirms the student's point: it is harder for black people to work with this software," said Virginia Dignum, an expert on the responsible use of artificial intelligence. Lucas Noldus, professor of Behavior, Information Technology and Innovation at Radboud University, comes to the same conclusion. "The conclusion is that black faces are detected the worst of all ethnicities. So then there is implicit discrimination. In this context, I can't explain it any other way."
Tessa Verhoef, associate professor at the Leiden Institute of Advanced Computer Science (LIACS), concurs. "It seems clear to me that it turns out that face detection for dark faces works significantly worse compared to other categories. That means there is an illegitimate discrimination."
Proctorio let it be known in a response that RTL News' research is flawed. For example, the images in the database used by the medium would not "simulate the experience of an exam candidate at a distance.
"None of the images were taken with a webcam. This study used the wrong material and tested a completely unrealistic data set. As a result, it is far from an accurate representation of how our system works, and its results, however interesting, should not be associated with our products and services," the company announced.
In a blog, Proctorio said that bias is a serious issue, but that there is no "significant bias. "Proctorio is actively addressing issues related to our face detection algorithms and ensuring a fair and inclusive online proctoring experience for all students. While the challenges remain, we recognize the need for ongoing improvements and are actively working toward that goal."
