A comparative study of violence risk assessment tools: A systematic review and metaregression analysis of 68 studies involving 25,980 participants

Abstract

There are a large number of structured instruments that assist in the assessment of antisocial, violent and sexual risk, and their use appears to be increasing in mental health and criminal justice settings. However, little is known about which commonly used instruments produce the highest rates of predictive validity, and whether overall rates of predictive validity differ by gender, ethnicity, outcome, and other study characteristics. We undertook a systematic review and meta-analysis of nine commonly used risk assessment instruments following PRISMA guidelines. We collected data from 68 studies based on 25,980 participants in 88 independent samples. For 54 of the samples, new tabular data was provided directly by authors. We used four outcome statistics to assess rates of predictive validity, and analyzed sources of heterogeneity using subgroup analysis and metaregression. A tool designed to detect violence risk in juveniles, the Structured Assessment of Violence Risk in Youth (SAVRY), produced the highest rates of predictive validity, while an instrument used to identify adults at risk for general offending, the Level of Service Inventory – Revised (LSI-R), and a personality scale commonly used for the purposes of risk assessment, the Psychopathy Checklist – Revised (PCL-R), produced the lowest. Instruments produced higher rates of predictive validity in older and in predominantly White samples. Risk assessment procedures and guidelines by mental health services and criminal justice systems may need review in light of these findings.

Introduction

Risk assessment tools assist in the identification and management of individuals at risk of harmful behaviour. Due to the potential utility of such tools, researchers have developed many risk assessment instruments, the manuals for which promise high rates of construct and predictive validity (Bonta, 2002). Recent meta-analyses have identified over 120 different risk assessment tools currently used in general and psychiatric settings (for a metareview, see Singh & Fazel, 2010). These measures range from internationally utilized tools such as the Historical, Clinical, Risk Management – 20 (HCR-20; Webster, Douglas, Eaves, & Hart, 1997) to locally developed and implemented risk measures such as the North Carolina Assessment of Risk (NCAR; Schwalbe, Fraser, Day, & Arnold, 2004). Given the large selection of tools available to general and secure hospitals and clinics, prisons, the courts, and other criminal justice settings, a central question is which measures have the highest rates of predictive accuracy. To date, no single risk assessment tool has been consistently shown to have superior ability to predict offending (Campbell et al., 2007, Gendreau et al., 2002, SBU, 2005, Walters, 2003), and several major uncertainties remain regarding the populations and settings in which risk assessments may be accurately used (Leistico et al., 2008, Guy et al., 2005, Schwalbe, 2008, Smith et al., 2009).

Such uncertainties are important given that risk assessment tools have been increasingly used to influence decisions regarding accessibility of inpatient and outpatient resources, civil commitment or preventative detention, parole and probation, and length of community supervision in many Western countries including the US (Cottle et al., 2001, Schwalbe, 2008), Canada (Gendreau et al., 1996, Hanson and Morton-Bourgon, 2007), UK (Kemshall, 2001, Khiroya et al., 2009), Sweden (SBU, 2005), Australia (Mercado & Ogloff, 2007), and New Zealand (Vess, 2008). Recent work has suggested that the influence of risk assessment tools appears to be growing in both general and forensic settings. For example, violence risk assessment is now recommended in clinical guidelines for the treatment of schizophrenia in the US and the UK (American Psychiatric Association, 2004, National Institute for Health and Clinical Excellence, 2009). In the US, risk assessment tools are used routinely in the mental health care systems of the majority of the 17 states that have civil commitment laws (Mercado & Ogloff, 2007). Recent studies in England have found that two thirds of mental health clinicians in general settings are using structured risk assessment forms (Higgins, Watts, Bindman, Slade & Thornicroft, 2005), as are clinicians working in over 70% of forensic psychiatric units (Khiroya et al., 2009). Risk measures are also being used with increasing regularity in both criminal and civil court cases in the US and the UK (DeMatteo and Edens, 2006, Young, 2009). The widespread, often legally required use of risk measures (Seto, 2005) necessitates the regular and high-quality review of the evidence base.

The research base on the predictive validity of risk assessment tools has expanded considerably; however, policymakers and clinicians continue to be faced with conflicting findings of primary and review literature on a number of central issues (Gendreau et al., 2000, Singh and Fazel, 2010). Key uncertainties include:

• Are there differences between the predictive validity of risk assessment instruments?

• Do risk assessment tools predict the likelihood of violence and offending with similar validity across demographic backgrounds?

• Do actuarial instruments or tools which employ structured clinical judgment produce higher rates of predictive validity?

There is contrasting evidence whether risk assessment tools are equally valid in men and women. Several recent reviews have found no difference in tool performance between the genders (e.g., Schwalbe, 2008, Smith et al., 2009). Schwalbe (2008) conducted a meta-analysis on the validity literature of risk assessment instruments adapted for use in juvenile justice systems, and found no differences in predictive validity based on gender. This finding was supported by a meta-analysis conducted by Smith, Cullen, and Latessa (2009), who found that the Level of Service Inventory – Revised (LSI-R) produced non-significantly different rates of predictive validity in men and women. In contrast, recent meta-analyses have found that the predictive validity of certain risk assessment tools is higher in juvenile men (Edens, Campbell & Weir, 2007) or in women (Leistico et al., 2008).

Another uncertainty is whether risk measures' predictive validity differs across ethnic backgrounds. There is evidence from primary studies and meta-analyses that risk assessment tools provide more accurate risk predictions for White participants than other ethnic backgrounds (Bhui, 1999, Edens et al., 2007, Långström, 2004, Leistico et al., 2008). This variation may be due to differences in the base rate of offending among individuals of different ethnicities (Federal Bureau of Investigation, 2002). These differences are seen in inpatient settings (Fujii et al., 2005, Hoptman et al., 1999, Lawson et al., 1984, McNiel and Binder, 1995, Wang and Diamon, 1999) and on discharge into the community (Lidz, Mulvey, & Gardner, 1993). Contrary evidence has been provided by reviews which have assessed the moderating influence of ethnicity on predictive validity rates in White, Black, Hispanic, Asian, and Aboriginal participants and have found no differences (Guy et al., 2005, Edens and Campbell, 2007, Schwalbe, 2007, Skeem et al., 2004).

Previous meta-analyses (e.g., Blair et al., 2008, Guy, 2008, Leistico et al., 2008) have found that participant age does not affect the predictive validity of risk assessment tools. However, epidemiological investigations and reviews (e.g., Gendreau et al., 1996) have found that younger age is a significant risk factor for offending. Therefore, we investigated the influence of age on predictive validity in the present meta-analysis.

Actuarial risk assessment tools estimate the likelihood of misconduct by assigning numerical values to risk factors associated with offending. These numbers are then combined using a statistical algorithm to translate an individual's total score into a probabilistic estimate of offending. The actuarial approach is an attempt to ensure that each individual is appraised using the same criteria and, in doing so, can be directly compared to others who have had the same tool administered regardless of who conducted the assessment.

The individual administering the assessment is thought to play a critical role when clinically based instruments are used. Broadly, clinical approaches to risk assessment can be dichotomized into unstructured clinical judgment and structured clinical judgment. Unstructured clinical judgment refers to a clinician's subjective prediction of whether an individual is likely to offend or not. No pre-specified set of risk factors is used to guide the clinician's analysis, which relies on professional experience for accuracy in assessing the likelihood of offending (Hanson, 1998). Recent reviews have suggested that this form of risk assessment has poor predictive validity (Daniels, 2005, Hanson and Morton-Bourgon, 2009). The poor performance of unstructured clinical judgment is thought to be a consequence of its reliance on subjective risk estimates that lack inter-rater and test–retest reliability (Hanson & Morton-Bourgon, 2009).

To increase construct validity and reliability, the authors of risk assessment tools developed new measures that adopt an approach known as structured clinical judgment (SCJ). In this approach, clinicians use empirically-based risk factors to guide their predictions of an individual's risk for offending (Douglas, Cox & Webster, 1999). Advocates of this approach believe that SCJ does more than simply assess risk, it provides information that can be used for treatment planning and risk management (Douglas et al., 1999, Heilbrun, 1997). While past reviews have provided evidence that actuarial tools produce higher rates of predictive validity than instruments which rely on structured clinical judgment (Hanson and Morton-Bourgon, 2009, Hanson and Morton-Bourgon, 2007), other researchers have presented evidence that both forms of risk assessment produce equally valid predictions (Guy, 2008). Due to this uncertainty, we investigated this issue in the present meta-analysis.

Secondary uncertainties in the field of risk assessment include whether study design characteristics such as study setting, prospective vs. retrospective design, or length of follow-up influence predictive validity (Singh & Fazel, 2010).

Prisons, psychiatric hospitals, courts, and the community are typical settings in most research in the forensic risk assessment literature (Bjørkly, 1995, DeMatteo and Edens, 2006, Edens, 2001, Bauer et al., 2003). Meta-analytic evidence regarding the moderating role of study setting on effect size varies (Leistico et al., 2008, Skeem et al., 2004). Some experts (e.g., Edens, Skeem, Cruise & Cauffman, 2001) suggest that differences in the accuracy of risk assessments may be attributed to the contextual differences in these study settings. Due to these differences, one measure may be superior to others in one study setting but not in another (Hanson & Morton-Bourgon, 2007).

Whether a study has a prospective or retrospective design may influence predictive validity findings. Being that the primary goal of risk assessment is to predict future offending, some researchers have stated that prospective research is not just appropriate, but necessary to establish a tool's predictive validity (Caldwell, Bogat & Davidson, 1988). However, strengths of retrospective study designs are that researchers do not have to wait for time to elapse before they investigate whether the studied individuals reoffended or not. This methodology is particularly useful with low base rate outcomes such as violent crime (Maden, 2001). Both actuarial and clinically based instruments can be used retrospectively. The latter tools can be scored using file information from sources such as psychological reports, institutional files, and/or court reports (de Vogel, de Ruiter, Hildebrand, Bos & van de Ven, 2004).

The evidence regarding the long-term efficacy of risk assessment tools is mixed. Using ROC analysis, Mossman (2000) concluded that accurate long-term predictions of violence were possible. In support, several recent meta-analyses have found that length of follow-up does not moderate effect size (Blair et al., 2008, Edens and Campbell, 2007, Edens et al., 2007, Schwalbe, 2007). Contrasting evidence has been found by reviews which have reported higher rates of predictive validity for studies with longer follow-up periods (Leistico et al., 2008, McCann, 2006, Smith et al., 2009). Other researchers have found tools to be valid only in the short-term and, even then, only at modest levels (Bauer et al., 2003, Sreenivasan et al., 2000). Given that studies often follow participants for different lengths of time and given that effect size may vary with time at risk, the role of this variable needs to be examined (Cottle, Lee, & Heilbrun, 2001). Very few tools have been validated for short follow-up time, such as hours, days or one to two weeks, which typically is a most relevant timeframe in clinical real-world decisions-making situations (SBU, 2005).

In summary, despite the increasing use and potential importance of risk assessment instruments, it is unclear which instruments have the highest rates of predictive validity, and whether these rates differ by important demographic and study design characteristics. We have therefore undertaken a systematic review and meta-analysis to explore rates of predictive validity in commonly used instruments and to assess the potential sources of heterogeneity outlined above.