Brief Rating of Aggression by Children and Adolescents (BRACHA): Development of a Tool for Assessing Risk of Inpatients' Aggressive Behavior

Scale Development

Before the present study, we developed a 67-item list of potential risk factors for physical or verbal aggression during hospitalization. We selected the 67 items from risk factors for aggression identified in previous studies^{2,3,7,13,16,24} and from risk and protective factors suggested by a committee of psychiatric clinicians (including psychiatrists, nurses, and social workers) at Cincinnati Children's Hospital Medical Center (CCHMC).

We piloted the 67 items with 289 patients in a quality-of-care project, to select the strongest risk indicators for aggression and violence. (As a quality-of-care project, this pilot study was deemed exempt by the CCHMC Institutional Review Board [IRB].) Forty percent (115/289) of the patients exhibited some form of aggression, and 14 of the 67 items were significantly associated (p < .05) with aggression. (More information about the pilot phase is available from the first author upon request.) Despite the lack of significance, two historical questions on sexual and physical abuse were retained in a second version of the scale, because clinical experience and some of the literature have identified maltreatment as a risk factor for aggression.³ Thus, the versions of the BRACHA evaluated in the current study consisted of 16 questions with yes or no responses (Table 1), plus easily obtained demographic data (birth date, sex, race, ethnicity, living arrangements, and type of insurance, a proxy for socioeconomic status) that we thought might be related to aggression (Table 2). Before continuation of the project, the CCHMC IRB reviewed the study again and reapproved it with a waiver of signed consent. Specific consent for BRACHA administration or OAS scoring was not required of patients or their guardians, because these instruments had become part of routine clinical care, and clinicians were using the instruments to make treatment and management decisions.

Procedure

Licensed psychiatric social workers administered the BRACHA to all patients during the emergency department (ED) admission process at CCHMC. Social workers received training on the instrument using live patient interviews of children, adolescents, and their guardians. Interviewers obtained answers to BRACHA questions primarily from the patients' parents or guardians, although information sometimes also came from patients or collateral sources.

The hospital where patients underwent treatment has 67 inpatient psychiatric beds divided into four units. The children and adolescents were placed on separate units divided by age (3–7 years, 8–11 years, and 12+ years). Before admission and bed assignment, the patients were classified into three risk groups based on their 16-item scores. Some adolescent patients in the highest risk category (those with scores of ≥12 of a possible 16) were placed on a safety unit that had higher levels of staffing. For safety reasons, unit staff members were not masked to patients' risk categories.

Outcome Measures

Hospital staff members used the Overt Aggression Scale (OAS)²⁵ to record inpatients' behavioral outcomes. The OAS characterizes an individual's verbal and physical aggression toward self, other persons, and objects. Using the OAS, a rater records the presence or absence of four categories of aggression: verbal aggression, physical aggression against self, physical aggression against other people, and physical aggression against objects. If an incident of aggression occurs, the rater assigns a score on a four-point severity scale anchored with behavioral descriptors. The OAS was developed for children and adults in the original studies, which established reliability with intraclass correlation coefficients (ICC) of 0.50 for verbal aggression and 0.81–1.00 for physical aggression.²⁵ A subsequent study confirmed the usefulness and appropriateness of the OAS for assessing aggression on a child psychiatric unit.²⁶ The OAS has been used in child and adolescent inpatient studies^27,28 and adult inpatient studies²⁹ to record and measure aggression.

In this article, we use the term aggression to designate any threatening verbal or physical behavior toward self, other people, or objects that would generate a score of 1 or higher on any subscale of the OAS. Examples of behavior that would earn a score of 1 for aggression include “shout angrily,” “hit self,” or “slam door.” We use the term violence to refer to actions that would generate a score of 1 or above on the “physical aggression toward other people” subscale of the OAS. Examples of behavior that would earn a score of 1 for violence include “make threatening gesture” or “swing at people.”²⁵

Hospital staff members were trained to use the OAS with vignettes. An OAS was completed for each patient after day (7 a.m.–3 p.m.) and evening (3 p.m.–11 p.m.) shifts and also after any aggressive episode. (OAS ratings were not obtained on the overnight shift because the expected incidence of aggression during this time was low.) At discharge, board-certified child and adolescent psychiatrists working on the inpatient units provided diagnoses for each patient based on Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR)³⁰ criteria using all available clinical information (interviews, historical information, observation, psychological testing, and collateral information from caregivers and outside treatment providers).

Sample

This study reports on findings for patients admitted during a six-month period beginning April 22, 2006, the first day the 16-item BRACHA questionnaire went into use. Data collection ended on October 31, 2006. During this period, CCHMC admitted 458 children and adolescents to psychiatric units. Nineteen patients were dropped from the analyses because of missing information about their living arrangements. Of the 145 patients for whom staff members documented aggressive episodes, 120 (82.8%) had their first aggressive incidents in the first six days of hospitalization. Based on this finding and on the clinical reality that most psychiatric inpatients have relatively short durations of stay, we limited analyses to aggression occurring in the first six hospital days.

During a six-day hospitalization, staff members should have completed at least 12 OASs (2 per day) for each patient. However, not all the OASs were completed. To avoid bias that might arise from having multiple missing OASs, we excluded from our analyses those patients who were missing six or more OASs. We chose six as the exclusion threshold, because during a six-day stay, a maximum of six OASs could be missing for legitimate reasons. For example, a patient who arrived at the hospital late at night, had weekend passes, or had an early morning discharge would not have an OAS for these periods. Twenty-one patients were excluded because six or more OAS ratings were missing, leaving a final sample of 418 patients. We found no evidence that the BRACHA 0.8 scores of the 21 excluded subjects differed from the scores of the other 418 subjects (p = .23, Wilcoxon two-sample test). Patients ranged in age from 3.5 to 19.1 years, with a mean ± SD age of 13.5 ± 3.4 years. One hundred ninety-one (45.7%) patients in the sample were female. Most patients (n = 258, 61.7%) were Caucasian (Table 2). All 418 patients in the sample had been admitted through the ED at CCHMC. No patient underwent more than one admission during the study period.

Data Collection and Analysis

All BRACHA and OAS data were entered by TeleForm. We evaluated each of the 16 questionnaire items and the patient's demographic information as potential predictors of aggression and violence (as previously defined). We treated age as a continuous variable, because inspection of the receiver operating characteristic (ROC) curve revealed a gradually decreasing slope. Data recording forms included several options concerning living arrangements (Table 2). Though studies indicate that caregiver burden in single-parent homes is higher than in two-parent households, we found that differences in aggression rates between patients who lived with one versus two biological parents did not differ significantly (χ² = 5.06, df = 1, p = .024) after taking multiple comparisons into consideration. We found a clearer difference, however, when we compared aggression rates of patients who lived with at least one biological parent with aggression rates for patients in other living situations (χ² = 8.88, df = 1, p < .0029). This being the case, we treated living arrangement as a dichotomous variable in subsequent data analyses (that is, living with one or two biological parent(s) = 1, other living arrangements = 0). Literature indicating that separation from a parent and lack of parental support are predictive of aggressive behavior supports this approach.^13,31 We used the chi-square and Fisher's exact tests to learn whether sex, race, and various diagnoses predicted aggression or violence.

Psychiatric Diagnoses

Ninety-eight percent of the patients (n = 410) had at least one primary or secondary diagnosis in the following categories: anxiety disorders and adjustment disorders (28.5%, n = 119), attention deficit hyperactivity disorder and impulse control disorders (37.1%, n = 155), disruptive behavior disorders (24.9%, n = 104), mood disorders (66.%, n = 276), developmental disorders (3.6%, n = 15), schizophrenia and other psychotic disorders (14.6%, n = 61), and mental retardation (4.3%, n = 18) (Table 3). Anxiety disorders and adjustment disorders were categorized together, since they often overlap and are difficult to distinguish during hospitalizations after acute stressors. The remaining eight patients had a substance use disorder, an eating disorder, or both.

Aggressive Versus Nonaggressive Patients

Table 2 summarizes relationships between demographic factors and occurrence of any aggression. During the first six days of hospitalization, 28.7 percent (120/418) of the patients committed a total of 292 aggressive acts. Of the 292 incidents, 102 (34.9%) were verbal, 81 (27.7%) were toward objects, 63 (21.6%) were toward others, and 46 (15.8%) were toward self. Forty (89%) of those patients who engaged in aggression toward self also engaged in other types of aggression, which suggests that in the hospital setting, harm to self shares some common risk factors with outwardly directed aggression. Patients with diagnoses of attention deficit hyperactivity disorder or other impulse control disorders were more likely to be aggressive (p < .0001), as were patients with disruptive behavior disorders (p = .0013). Associations with other conditions (e.g., mood disorders) were not significant after adjustment of significance levels for multiple comparisons (Table 3).

All questionnaire items were significantly associated with aggression except for the two questions about personal abuse history (questions 4 and 5; Table 1). The 16 items had good internal consistency (Cronbach's α = 0.818), and internal consistency improved when the two nonpredictive items were deleted (Cronbach's α = 0.837). Although these values suggest that the scale is unidimensional, we conducted factor analyses on the 14 predictive items. Exploratory factor analysis with an orthogonal or varimax rotation indicated a three-factor solution: personal history (questions 1–3, 6, and 12–16); past violence (questions 9–11); and impulsiveness in the ED (questions 7 and 8). Kaiser's Measure of Sampling Adequacy (MSA) was 0.82, a good indicator that the data are appropriate for the common factor model. Internal consistency for each factor was good, with Cronbach's α = 0.79 for Factor 1, α = 0.84 for Factor 2, and α = 0.78 for Factor 3.

Evaluation of Risk Assessment Potential

As Table 2 indicates, several demographic variables were clearly associated with aggression during hospitalization. Thus, an important threshold problem in evaluating our questionnaire was to see whether the responses to the 14 items added any predictive power to what one might conclude from demographic data that are quickly and reliably obtainable. We decided not to evaluate race as a potential predictive item, in part because we thought that apparent race-related differences in aggression rates might reflect race-related differences in interpreting and scoring aggressive incidents. Using stepwise, backward-elimination logistic regression models, we evaluated the potential predictive role of the remaining demographic factors. Results showed that when age and responses to the 14-item scale were included, the other three demographic factors added no predictive value. We therefore evaluated prediction models that used only patients' ages and questionnaire results to gauge the risk of aggression or violence.

The following equation describes the BRACHA 0.7 prediction model: (1)

As Equation 1 shows, one computes a BRACHA 0.7 score by subtracting a patient's age from 20 plus the total number of questionnaire items answered yes. (The constant 20 simply assures that the total score will be positive.) BRACHA 0.7 has 14 items, and the patients' age range is 3 to 19 years. BRACHA 0.7 scores thus range from 1 to 31, with aggression risk increasing as the score increases. Notice that BRACHA 0.7 gives roughly equal predictive weight to a patient's age (which has a range of 16 years) and the 14-item score; age is subtracted in the computation, because age is inversely related to likelihood of aggression.

The second model, BRACHA 0.8, takes advantage of the three component factors discussed above, while aiming to avoid overfitting that might result from using β-weights generated from a logistic regression using our 418-patient sample. One calculates a BRACHA 0.8 score thus: (2)

Equation 2 implies that a yes answer to a given questionnaire item contributes to the prediction equation in inverse proportion to the number of items in the factor of which the given item is an element. For example, Item 1 is one of nine elements of Factor 1, and so a yes in response to Item 1 adds five-ninths of a point to the total score. Similarly, a yes answer for one of the two items making up Factor 3 adds 2 points to the total score. Scores on the BRACHA 0.8 can range from 1 to 32. One multiples the three factor sums by 5 so that, as with BRACHA 0.7, age and questionnaire answers contribute roughly equally to the total risk assessment score.

We used receiver operating characteristic (ROC) analysis to evaluate the performance of BRACHA 0.7 and 0.8 and to compare their accuracy with two simpler predictors: patients' ages and patients' total scores on the 14 items from the questionnaire. ROC analysis is the appropriate method for evaluating the performance of a risk assessment tool for which possible outcomes are arrayed along a continuous or ordinal scale.³² A ROC graph plots test sensitivity (equal to the true-positive rate, tpr) as a function of the false-positive rate (fpr = 1 − specificity). The area under the ROC curve (AUC) is a useful summary measure of test performance. In this context, AUC equals the probability that the instrument will assign a patient randomly chosen from the aggressive subgroup a higher score (i.e., a higher likelihood of aggression) than a patient randomly chosen from the nonaggressive subgroup. Given such a pair of patients, an instrument with no discriminative power would perform no better than flipping a coin to classify them and would have an AUC of 0.5. An instrument that always assigned a higher score to the aggressive patient would have an AUC of 1.0. Adult studies examining performance of actuarial risk assessment instruments typically report AUCs of 0.65 to 0.80.^19,32 A recent study of actuarial risk assessments for violent offending in 74 teenage outpatients reported an AUC range of 0.71 to 0.73.²⁴

We calculated AUCs for BRACHA 0.7 and 0.8 by using ROCKIT software.³³ This package permits comparison of correlated ROC curves, an important consideration when, as in the present study, accuracy parameters are derived from applying two or more assessment methods to the same subjects. Table 4 describes the performance of BRACHA 0.7 and 0.8 in assessing risk of aggression and violence and shows pair-wise comparisons of these scales' performances with what one would expect by simply using age or the 14 items by themselves as risk assessment tools. Figures 1 and 2 show the ROC curves for age alone, the 14 items, and BRACHA 0.7, and 0.8.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Receiver operating characteristic (ROC) curves for prediction of any form of aggression.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Receiver operating characteristic (ROC) curves for prediction of interpersonal violence.

In assessing risk of aggression toward others, age alone was a good predictor (AUC = 0.7691) but not significantly greater than the 14-item score (AUC = 0.7154). However, combining age and the 14 items yielded predictive formulae with AUCs above 0.8 and, in the case of BRACHA 0.8, accuracy was significantly higher than the 14-item score (p < .01) or age alone (p < .05).

AUC values provide a global measure of discrimination power, but they do not directly help clinicians grasp the practical effect of using an assessment method. One way to appreciate what an assessment tool might do is to suppose that a practitioner picked a single cutoff score to differentiate between patients at high and low risk for aggressive or violent behavior. Methods of cutoff selection include maximizing sensitivity plus specificity or utility (benefits minus costs). However, the former method is insensitive to base rate considerations, and the latter method requires establishing precise values for true positives and true negatives and precise costs for false positives and false negatives. Establishing these precise amounts usually is impossible, but a reasonable, practicable alternative is to find and evaluate the consequences of using the cutoff that yields maximum diagnostic information (MDI).

To select the MDI cutoff for any form of aggression, we used the base rate in our sample (120/418 = 0.287), binormal accuracy parameters generated by the ROCKIT software, and the cutoff location procedure described by Somoza and Mossman.³⁴ The (fpr, tpr) pair that maximized diagnostic information was (0.431, 0.802; Fig. 1), which corresponds to a BRACHA 0.8 score of 13. This operating point would divide the patients into two groups: a 224-member higher-risk group with a 43 percent rate of aggression (that is, a group containing 96 actually violent and 128 actually nonviolent patients) and a 194-member lower-risk group with a 12 percent rate of aggression (170 actually nonviolent and 24 actually violent patients). Using the same process to find the MDI cutoff for violence (base rate = 63/418 = 0.151) led to a MDI cutoff at (fpr, tpr) = (0.327, 0.822; Fig. 2), which corresponds to a BRACHA 0.8 score of 14. This operating point would divide patients into a 168-member higher risk group in which 52 (31%) patients were actually violent and 116 were not, and a 250-member, lower risk group of whom 9 (3.6%) patients were actually violent and 241 were not.

Readers should note that, in this article, we have used the MDI cutoff primarily for demonstration purposes. An interesting consequence of using either 13 or 14 as a cutoff for the BRACHA 0.8 is that children ages 7 and under automatically fall into the high-risk group. While such a categorization reflects the strong inverse relationship between age and aggression risk, using a single cutoff for all ages might not be useful in actual clinical contexts.