The Classification of Violence Risk

The Classification of Violence Risk (COVR®) was developed with the goal of offering clinicians an actuarial ‘‘tool’’ to assist in their predictive decision making. The COVR is an interactive software program designed to estimate the risk that an acute psychiatric patient will be violent to others over the next several months after discharge from the hospital. Using a laptop or desktop computer, the COVR guides the evaluator through a brief chart review and a 5–10 minute interview with the patient. After the requested information has been entered, the COVR generates a report that contains a statistically valid estimate of the patient’s violence risk, including the confidence interval for that estimate and a list of the risk factors that the COVR took into account to produce the estimate. In this article, we describe the development of the COVR and address several key issues in its administration. Detailed descriptions of the research constructing and validating the software and the statistical model on which it rests can be found in the work of Monahan et al. (2001, 2005b).

The Development of the COVR

The empirical foundation of the COVR was developed in eight stages over an 18 year period.

Stage One: Identifying Gaps in Methodology

When we began our work, almost all existing studies of violence risk assessment suffered from one or more methodological problems. The studies (a) considered a constricted range of risk factors, often a few demographic variables or scores on a psychological test, (b) employed weak criterion measures of violence, usually relying solely on arrest, (c) studied a narrow segment of the patient population, typically males with a history of prior violence, and (d) were conducted at a single site (Monahan & Steadman, 1994a). Based upon this critical examination of existing work, the authors designed the MacArthur Violence Risk Assessment Study with the aim of overcoming the identified methodological obstacles (Monahan & Steadman, 1994b). To overcome the methodological problems found in existing studies, the authors (a) studied a large and diverse array of risk factors, (b) triangulated their outcome measurement of violence, adding patient self-report and the report of a collateral informant to data from official police and hospital records, (c) studied both men and women, regardless of whether they had a history of violence, and (d) conducted the study at three sites rather than at a single site.

Stage Two: Selecting Promising Risk Factors

Studies have suggested that a number of variables might be robust risk factors for violence among people with a mental disorder. The authors assessed personal factors (e.g. demographic and personality variables), historical factors (e.g. past violence, mental disorder), contextual factors (e.g. social support, social networks), and clinical factors (e.g. diagnosis, specific symptoms). Next, the authors chose what they believed to be the best of the existing measures of these variables, and they commissioned the development of a necessary measure where no adequate measure to assess a variable was available (e.g. the Novaco Anger Scale; Novaco, 1994).

Stage Three: Using Tree-Based Methods

The authors developed violence risk assessment models based on a ‘‘classification tree’’ method rather than on the usual linear regression method (Gardner, Lidz, Mulvey, & Shaw, 1996). A classification tree approach prioritizes an interactive and contingent model of violence—one that allows many different combinations of risk factors to classify an individual at a given level of risk (Breiman, Friedman, Olshen, & Stone, 1984). The particular questions to be asked in any assessment grounded in this approach depend on the answers given to prior questions. Factors that are relevant to the risk assessment of one individual may not be relevant to the risk assessment of another individual. This approach contrasts with a regression approach in which a common set of questions is asked of everyone being assessed and every answer is weighted to produce a score that can be used for purposes of categorization.

Stage Four: Creating Different Cut-offs for High and Low Risk

Rather than relying on the standard single threshold for distinguishing among participants, the authors decided to employ two thresholds—one for identifying high-risk individuals and one for identifying low-risk individuals. It was assumed that inevitably there will be individuals who fall between these two thresholds—individuals for whom any actuarial prediction scheme is incapable of making an adequate assessment of high or low risk. The degree of risk presented by these intermediate individuals cannot be statistically distinguished from the base rate of the sample as a whole; therefore, we refer to these individuals as constituting an ‘‘average-risk’’ group.

Stage Five: Repeating the Classification Tree

To increase the predictive accuracy of a classification tree, those individuals designated as ‘‘average risk’’ were reanalyzed. That is, all of the participants who were not classified into groups designated as either ‘‘high’’ or ‘‘low’’ risk in the standard classification tree model were pooled together and reanalyzed. The reason for reanalyzing these data was to determine whether the individuals who were not classified in the first iteration of the analysis might be different in some significant ways from the individuals who were classified, with the full set of risk factors available to generate a new classification tree specifically for those individuals who were not already classified as high or low risk. This resulting classification tree model is referred to as an ‘‘iterative classification tree’’ or ICT (Steadman et al., 2000).

Stage Six: Combining Multiple Risk Estimates

An important characteristic of the classification tree methodology is that variables entered initially into the tree carry more weight in determining the risk group to which an individual is assigned. Therefore, as a final step, the authors estimated several different risk assessment models in an attempt to obtain multiple risk assessments for each individual. That is, different risk factors were chosen to be the lead variable upon which a classification tree was constructed. In attempting to combine these multiple risk estimates, the authors began to conceive of each separate risk estimate as an indicator of the underlying construct of interest—violence risk. The basic idea was that individuals who scored in the high-risk category on many classification trees were more likely to be violent than individuals who scored in the high-risk category on fewer classification trees. Analogously, individuals who scored in the low-risk category on many classification trees were less likely to be violent than individuals who scored in the low-risk category on fewer classification trees (Banks et al., 2004). The result of this ‘‘multiple iterative classification tree’’ procedure was to place each patient into one of five risk groups, whose likelihood of violence to others over the next several months ranged from 1% to 76%.

Stage Seven: Developing the COVR Software

The multiple ICT models that were constructed had an impressive capacity to identify individuals with differing levels of violence risk; however, they were also very computationally intensive and not suited to paper-and-pencil administration. As a result, the authors obtained a Small Business Innovation Research grant from the National Institute of Mental Health to develop user-friendly software that could be employed to classify patients at differing levels of violence risk. The software is capable of assessing 40 risk factors (listed by Monahan et al., 2001, Table 6.3), but in any given case assesses only the number of risk factors necessary to classify the patient’s violence risk according to the multiple ICT procedure.

Stage Eight: Prospectively Validating the COVR Software

Next, the National Institute of Mental Health grant allowed the authors not only to develop software for violence risk assessment, but also to validate that software prospectively. The authors administered the newly developed software, entitled the COVR, to independent samples of acute civil inpatients at two sites. Patients classified by the software as at high or low risk of violence were followed in the community for 20 weeks after discharge. Expected rates of violence in the low- and high-risk groups were 1 and 64%, respectively. Observed rates of violence in the low- and high-risk groups were 9 and 35%, respectively, when a strict definition of violence was used, and 9 and 49%, respectively, when a slightly more inclusive definition of violence was used. These results indicated that software incorporating the multiple ICT models may be helpful to clinicians who are faced with making decisions about discharge planning for acute civil inpatients (Monahan et al., 2005a).

Key Issues in the Administration of the COVR Software

Here we address four issues that are frequently raised in discussions of the COVR software: reliance on self-report information, clinical interpretation of the risk estimate, communicating the risk estimate, and implications of the software for risk management.

Self-Report: Can it be Believed?

In both the original research in which the COVR’s multiple ICT methodology was developed and in the subsequent research in which this methodology was validated, the authors operated under the protection of a Federal Confidentiality Certificate and the patients studied were made aware of this protection. The certificate meant that most disclosures that the examinees made to the examiners were not discoverable in court and did not have to be reported to the police. In addition, in both the development study and the validation study, the examinees were notified that information from their police and hospital records was being collected, as well as information from interviewing a collateral informant who knew them well, to verify the information given by the examinee. Both of these components—the guarantee of confidentiality and the reliance on multiple information sources—may have encouraged examinees to provide ‘‘more truthful’’ information.

However, in the real world of clinical practice in which the COVR will be used, Federal Confidentiality Certificates will not be obtainable and police records and collateral informants will not be routinely available. Will examinees be as forthcoming and honest in answering questions when the COVR is used as a tool to make actual decisions on the nature or venue of their care as they were when the COVR was being constructed or validated and the answers had no personal impact? More to the point, what is the clinician to do when he or she has reason to doubt the truthfulness of an answer that a client gives to a question that the COVR presents?

Many examinees’ answers, of course, cannot be ‘‘verified,’’ because they relate to events in their interior lives (e.g., Does the patient really daydream often about harming someone?). Other answers are, in principle, capable of verification, but could be verified only with great difficulty in usual clinical situations (e.g., Were the patient’s parents arrested when the patient was a child?). Still other answers may be verifiable, to a greater or lesser degree, by recourse to data in the patient’s existing hospital chart or outpatient record. What is the clinician to do when a patient’s answer to a question posed by the COVR is contradicted or called into question by other data sources? Consider the following examples.

• When asked about using alcohol or other drugs, Patient A answers in the negative, yet the chart indicates that when he was admitted to the hospital on the previous evening, he had a blood alcohol level of. 30 or had tested positive for cocaine in his urine.

• When asked about past arrests, Patient B denies ever having been arrested, yet the file contains previous evaluations for competence to stand trial on felony charges.

• When asked whether she was abused as a child, Patient C answers ‘‘Never,’’ but there is a notation in the chart that the patient’s mother and sister stated that the mother’s boyfriend repeatedly sexually assaulted the patient when she was a girl.

Four courses of action are theoretically possible in situations involving conflicting information. First, the clinician could simply enter into the COVR the patient’s answers as given, even if the clinician were convinced that the examinee was being untruthful.

Second, the clinician could enter into the COVR his or her best judgment as to the factually correct answer to the question asked. For example, the clinician could choose to credit the toxicology report indicating recent drug abuse over the patient’s self-reported denial of drug abuse, and enter ‘‘yes’’ to the appropriate drug abuse question.

Third, the clinician familiar with the contents of the chart could confront the patient when apparent discrepancies arise between chart information and the patient’s answers. For example, the clinician could say to the patient who denied ever having been arrested, ‘‘I have a problem. You say you’ve never been arrested, but in your hospital record there’s a competence report that says you were arrested for assault twice last year and once the year before. What about this?’’

Finally, the clinician, on being convinced from information available in the record that the examinee was being untruthful in his or her answers, could simply terminate the administration of the COVR and arrive at a risk estimate without the aid of this actuarial tool. For example, the clinician could base his or her clinical risk estimate entirely on information available in the chart or from other data sources that do not rely on the patient’s unreliable self-reports.

Which of these options is recommended? It is important to emphasize that the COVR was constructed and validated using a variant of the third option described above. Although information obtained from a collateral informant is never revealed to the examinee, the examinee would be confronted on any apparent inconsistencies between his or her answers and information contained in the hospital chart.

The first option of simply entering into the program the examinee’s answers as given, even if the clinician is convinced that the examinee is being untruthful, is clinically and ethically inappropriate. If the clinician does not clearly note the patient’s apparent untruthfulness in the report that accompanies the COVR score, the clinician would knowingly be basing a risk estimate on information that, in his or her best judgment, is invalid, without warning potential users of the estimate of its uncertain foundation. Moreover, if the clinician does clearly note the patient’s apparent untruthfulness in the accompanying report, the clinician would, in effect, be telling the reader to disregard the risk estimate that the clinician had just offered.

The second option of the clinician entering into the COVR his or her best judgment as to the factually correct answer to the question asked, rather than the answer that the patient provided, is problematic for two reasons. First, it makes no attempt to determine whether there is an explanation for the apparent discrepancy that would indicate that the patient’s account is actually correct (e.g., someone else’s toxicology laboratory slip was mistakenly placed in the patient’s chart). Second, it varies from the procedures used to construct and validate the COVR, hence threatening the validity of the resulting estimate of risk. In the latter regard, to use clinician-generated answers rather than patient-generated answers raises the question of whether a clinician can accurately detect a patient’s deception when it occurs, without at the same time erroneously considering many true responses to be deceptive. On the other hand, as mentioned earlier, the guarantee of confidentiality and the reliance on multiple information sources that was used in developing and validating the COVR may have encouraged patients to provide us with ‘‘true’’ information, not just with ‘‘self-report’’ information. If this is the case, one can argue that, in clinical practice, entering information believed to be ‘‘true’’ is actually the more empirically appropriate strategy.

Our recommendation for the user of the COVR is the third option—the examiner familiar with the contents of the chart should confront the patient when apparent discrepancies arise between the chart information and the patient’s answers. If the discrepancy is satisfactorily resolved, the clinician would then enter into the COVR the answer that the patient gives. If the discrepancy is not satisfactorily resolved (e.g. if, after confrontation, the clinician still credits a recent toxicology report indicating drug abuse over the patient’s denial of ever having used drugs), then the clinician would enter ‘‘missing’’ for that piece of data, rather than entering either the patient’s self-report or the clinician’s own judgment about the accurate scoring of this item, being sure to note this action in an accompanying report. Our rationale for this recommendation is that the COVR contains 10 classification tree models but can produce a reliable estimate of risk using only five models (Banks et al., 2004). As long as at least five models contain no missing data due to the clinician disbelieving a patient’s answer (or for any other reason), the COVR will operate as designed. If more than five models contain missing data, the COVR will not produce an estimate of risk and the clinician will have to arrive at a risk estimate without the aid of this actuarial tool (i.e. choose the fourth option above).

Clinical Interpretation: Is Professional Review Necessary?

Several issues may arise during the administration of the COVR that may bring into question the validity of the examinee’s responses. As a result, it is important to keep in mind that the COVR software is useful in informing, but not replacing, clinical decision making regarding risk assessment. The authors recommend a two-phased violence risk assessment procedure, in which a patient is first administered the COVR, and then the risk estimate generated by the COVR is reviewed by the clinician ultimately responsible for making the risk assessment—who may or may not have been the clinician who administered the COVR—in the context of additional relevant information gathered from clinical interviews, significant others, and/or available records. Although clinical review would not revise or ‘‘adjust’’ the actuarial risk score produced by the COVR, and could in principle either improve or lessen predictive accuracy as compared with relying solely on an unreviewed COVR score, it seems essential to encourage such a review. Not only does the application of clinical judgment represent the standard of care for legal purposes, but data in addition to those collected by the COVR may properly impact violence risk assessment and should not be ignored (Monahan et al., 2001). For example, a low-risk patient who makes a convincing threat of physical harm to another person might be dealt with more cautiously than his or her COVR score would suggest.

Risk Communication: What are the Options?

Much recent research has addressed the manner in which risk estimates are communicated to decision makers (Heilbrun et al., 2004; Monahan et al., 2002; Monahan & Steadman, 1996; Slovic, Monahan, & MacGregor, 2000). Three format options for communicating risk are frequently discussed: (a) probability, (b) frequency, and (c) categories. Each of these three formats appears to have advantages and disadvantages. Because the field of risk communication is very new, no professional consensus has emerged on a standard way of communicating risk estimates. For this reason, the COVR report generates these multiple formats and then allows the user to choose the format in which risk estimates are conveyed: as a probability, as a frequency, as a category, or as some combination of these three formats.

In the probability format, risk is communicated in terms of percentages for the 95% confidence interval and the best point estimate. For example, the first of five levels of risk is communicated in the probability format as ‘‘The likelihood that the patient will commit a violent act toward another person in the next several months is estimated to be between 0% and 2%, with a best estimate of 1%.’’ For the other four levels of risk, the probabilities are expressed as ‘‘between 5% and 11%, with a best estimate of 8%,’’ ‘‘between 20% and 32%, with a best estimate of 26%,’’ ‘‘between 46% and 65%, with a best estimate of 56%,’’ and ‘‘between 65% and 86%, with a best estimate of 76%.’’

In the frequency format, risk is communicated in terms of whole numbers for the 95% confidence interval and the best point estimate. For example, the first of five levels of risk is communicated in the frequency format as ‘‘Of every 100 people like the patient, between 0 and 2 are estimated to commit a violent act toward another person in the next several months, with a best estimate of 1 person.’’ The other four levels of risk are analogously communicated using whole numbers.

In the categorical format, risk is communicated in terms of the group or class into which an individual is estimated to fall. The default options are the following.

• Category 1: very low risk [corresponding to a risk of 1%/1 of 100]

• Category 2: low risk [corresponding to a risk of 8%/8 of 100]

• Category 3: average risk [corresponding to a risk of 26%/26 of 100]

• Category 4: high risk [corresponding to a risk of 56%/56 of 100]

• Category 5: very high risk [corresponding to a risk of 76%/76 of 100].

Note that the labels given to each of the five categories are merely illustrative and can be defined by the user. For example, Category 5 could be re-labeled as ‘‘Discharge must be approved by Dr Smith.’’

Risk Management: How Does it Relate to Risk Assessment?

It must be emphasized that a judgment about the degree of risk posed by a patient is merely the beginning of the process of risk management. For a patient judged to present a lower risk of violence to others, there may be reasons, nonetheless, to keep him or her hospitalized, including risk to the patient himself or herself, need for further diagnostic assessment or treatment trials, and extended discharge planning. Conversely, even higher-risk patients may reasonably warrant hospital discharge (e.g. when appropriate plans have been put in place to manage risk in the community, and the gains to the patient and others from discharge justify the risk taken). The latter is especially important given that the COVR, like most prediction instruments developed to date, relies strongly (although not exclusively) on historical variables (e.g. history of arrests) that will not change with treatment. Although these may be valid indicators of risk, they are insensitive to changes in the patient’s clinical condition that may indicate that the risk is now easier to manage. Thus, placement of a patient into a given level of violence risk should not be confounded with the decision as to whether he or she should be retained in the hospital or discharged (Monahan & Appelbaum, 2000; Monahan & Silver, 2003). For example, delusions do not play a major role as risk factors in the COVR. Yet we have elsewhere cautioned against ignoring delusions in a given case (Appelbaum, Robbins, & Monahan, 2000):

Conclusion

We cannot stress strongly enough that the COVR software was constructed and has been validated only on samples of psychiatric inpatients in acute facilities in the United States who would soon be discharged into the community. Whether the validity of the model can be generalized to other people (e.g. people without mental disorder, people outside the United States) or to other settings (e.g. outpatient settings, criminal justice settings) remains to be determined empirically. Until such evidence is available—and a number of projects are underway to generate the required evidence—use of the model should be restricted to acute inpatient populations. It is also unclear whether repeated administration of the software to the same patients will lead to attempts to ‘‘game’’ the system by providing answers intended to understate the degree of risk, and if so what impact this will have on the validity of risk estimates. Answers to this question will await studies using the software in actual clinical settings. At this point, however, the Classification of Violence Risk (Monahan et al., 2005a), incorporating the multiple ICT model, may be helpful to clinicians in the United States who are faced with making decisions about discharge planning for acutely hospitalized civil patients.