Response to questions about doubts about the use of a particular risk assessment instrument in probation because of the risk of ethnic profiling

Question 1
Are you aware of the report that the algorithms used by the probation department may be causing ethnic profiling? (1)

Answer 1
Yes.

Question 2
Has the Ministry of Justice and Security or the probation service itself tested whether the algorithm mentioned in the cited article could indirectly lead to ethnic profiling? If not, why not and are you willing to still test this thoroughly?

Answer 2
During the development of the Dutch version of the instrument in question, the OxRec (Oxford Risc of Recidivism Tool) in 2017, the probation service set as a precondition from the outset that no aspects are recorded in the instrument regarding ethnicity. Therefore, the Dutch version also does not ask about the migration background of the suspect, where the English version does.

The cited article argues that including the neighborhood score may indirectly lead to ethnic profiling. Including the neighborhood score in the OxRec does not select on the basis of race, ethnicity, religion, or ancestry, but on the basis of the suspect's socioeconomic status. Indeed, scientific research shows that this affects recidivism risk.

Question 3
What is the importance of the algorithm in the set of instruments used by the probation officer to estimate recidivism risk? For example, is this algorithm used at the very end of the risk assessment to verify the assessment that the probation officer had already made himself or, on the contrary, is the algorithm an integral part of the final risk assessment?

Answer 3
The OxRec is an integral part of the probation worker's work. It supports the probation worker in assessing the likelihood of recidivism and preparing the probation recommendation. It is designed as a tool for this so that the probation worker does not overlook anything and critically reviews his own assessment.

Question 4
Is it true that the algorithm used by the probation department uses information about the zip code where a person lives, the crime rate in that neighborhood and the percentage of welfare payments? If so, why do you consider this desirable?

Answer 4
All factors measured in the OxRec have been validated using data from Dutch and Swedish litigants. It has therefore been investigated whether these factors indeed contribute to the estimation of the risk of recidivism. In the instrument, the postal code of the offender is indeed entered. Based on socioeconomic data from the Central Bureau of Statistics, the OxRec provides a neighborhood score. In fact, scientific research shows that this increases the predictive value of the instrument. In other words, it provides a more reliable estimate of recidivism risk. Furthermore, the score can also be plotted for clients without permanent residence or if they are in detention. The probation service has included the neighborhood score in the instrument in order to be able to provide targeted support and guidance to the offender in order to reduce the risk of recidivism.

Question 5
To what extent does the probation department itself have insight into the source code of the algorithm and thus into the operation of the algorithm? Can it itself see which components someone scores high on or does the algorithm produce only one outcome, for example: high risk, medium risk, low risk?

Answer 5
The probation department has insight into the algorithm used in the OxRec and how it was created. This is because the developers have been transparent in developing the OxRec. As a result, it is clear what steps were taken in the development and validation of the model. For example, a research protocol was submitted in advance and the extent to which it was deviated from is accounted for afterwards. Articles are peer reviewed and published open access so that they are accessible for all to read. This provides insight into which risk factors were chosen and which statistical models were used to calculate the outcome.

When using the OxRec, the outcome is not a high or low risk category. The model estimates the probability of recidivism where a probability score expressed in percentages is given. This is shown for both general recidivism and violent recidivism within 1 and 2 years. (2) It is thus nuanced, transparent and insightful for the probation worker.

Question 6
Is it true that the margins of error in the algorithm's calculations are significant? If so, do you consider this desirable given the impact it can have on a suspect? If not, can you demonstrate what the margins of error are in OxRec?

Answer 6
A risk assessment is an estimate in which one compares characteristics of an individual with a group average of a group of persons with similar characteristics. Thus, estimating the risk of recidivism as low does not mean that someone will definitely not recidivate, and estimating the risk as high does not mean that recidivism in an individual can be determined with certainty. So in this sense, when using risk assessment instruments, there is always a margin of uncertainty. It is important that this margin be as small as possible, but predicting human behavior, in this case recidivism, is very complex.

Scientific research has been conducted on how well the OxRec estimates the likelihood of recidivism. (3) This research shows that the accuracy of the instrument is within the range that can be expected from risk assessment instruments. There are several risk assessment instruments that estimate the likelihood of recidivism. The predictive value of the OxRec is not inferior to that of other instruments.

(1) Allegiance, June 26, 2020, "Algorithms used by probation provide ethnic profiling," https://www.trouw.nl/binnenland/algoritmes-gebruikt-door-reclassering-zorgen-voor-etnisch-profileren~b8918776/
(2) See also the article by the probation service in the Dutch Law Journal: Maas, M., Legters, E., Fazel, S. (2020). Professional and risk assessment tool hand in hand. Nederlands Juristenblad. Also available on the probation service's website: https://www.reclassering.nl/actueel/nieuws/artikel-reclassering-over-oxrec-in-nederlands-juristenblad.
(3) Fazel, S., Wolf, A., Vazquez-Montes, M.D.L.A. & T.R. Fanshawe, T.R. (2019) Prediction of violent reoffending in prisoners and individuals on probation: a Dutch validation study (OxRec)', Scientific Reports 2019, 9: 841.