A note on the multiplicative fairness score in the NIJ recidivism forecasting challenge

dc.contributor.authorMohler, George
dc.contributor.authorPorter, Michael D.
dc.contributor.departmentComputer Science, Luddy School of Informatics, Computing, and Engineering
dc.date.accessioned2024-04-03T11:07:24Z
dc.date.available2024-04-03T11:07:24Z
dc.date.issued2021
dc.description.abstractBackground: The 2021 NIJ recidivism forecasting challenge asks participants to construct predictive models of recidivism while balancing false positive rates across groups of Black and white individuals through a multiplicative fairness score. We investigate the performance of several models for forecasting 1-year recidivism and optimizing the NIJ multiplicative fairness metric. Methods: We consider standard linear and logistic regression, a penalized regression that optimizes a convex surrogate loss (that we show has an analytical solution), two post-processing techniques, linear regression with re-balanced data, a black-box general purpose optimizer applied directly to the NIJ metric and a gradient boosting machine learning approach. Results: For the set of models investigated, we find that a simple heuristic of truncating scores at the decision threshold (thus predicting no recidivism across the data) yields as good or better NIJ fairness scores on held out data compared to other, more sophisticated approaches. We also find that when the cutoff is further away from the base rate of recidivism, as is the case in the competition where the base rate is 0.29 and the cutoff is 0.5, then simply optimizing the mean square error gives nearly optimal NIJ fairness metric solutions. Conclusions: The multiplicative metric in the 2021 NIJ recidivism forecasting competition encourages solutions that simply optimize MSE and/or use truncation, therefore yielding trivial solutions that forecast no one will recidivate.
dc.eprint.versionFinal published version
dc.identifier.citationMohler G, Porter MD. A note on the multiplicative fairness score in the NIJ recidivism forecasting challenge. Crime Science. 2021;10(1):17. doi:10.1186/s40163-021-00152-x
dc.identifier.urihttps://hdl.handle.net/1805/39713
dc.language.isoen_US
dc.publisherSpringer Nature
dc.relation.isversionof10.1186/s40163-021-00152-x
dc.relation.journalCrime Science
dc.rightsAttribution 4.0 Internationalen
dc.rights.urihttp://creativecommons.org/licenses/by/4.0/
dc.sourcePublisher
dc.subjectRecidivism
dc.subjectMultiplicative metrics
dc.subjectPredictions
dc.titleA note on the multiplicative fairness score in the NIJ recidivism forecasting challenge
dc.typeArticle
Files
Original bundle
Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
Mohler2021Note-CCBY.pdf
Size:
994.83 KB
Format:
Adobe Portable Document Format
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.99 KB
Format:
Item-specific license agreed upon to submission
Description: