Exploring the Relationship Between Scoring Accuracy and Predictive Validity in Risk Assessment Using the Service Planning Instrument (SPIn)

Accurate scoring is an essential component of a risk assessment's reliability, which in turn contributes to its predictive validity. This study tested a new method for evaluating the accuracy of completed risk assessments, described as intra-rater consistency. Pairs of cross-consistency items in the Service Planning Instrument were used to identify scoring errors. Participants were 31,460 adults (20.5% female; 20.7% Indigenous) on community supervision in Alberta. The overall base rate of error was low; Indigenous participants had higher rates of error than non-Indigenous participants. In the full sample assessments with errors had slightly higher predictive accuracy than assessments without errors (error AUC = .70, no error AUC = .68). Similar results were found among non-Indigenous sub-groups, however, there were no significant differences in predictive accuracy based on errors among Indigenous participants. Sex- and race-based differences in predictive accuracy were observed in assessments with errors, but not in assessments without errors.