Commentary: Evidence-Based Practice and Forensic Psychiatry

I should state at the outset that the perspective and commentary that follow come from my experience as a judge whose court is a regular consumer of scientific evidence. The authors who have contributed to this issue of the Journal are experts in their fields. I, quite clearly, am not an expert in forensic science. It is inevitable, therefore, that I may have misapprehended some of what they have had to say, and I apologize in advance for any mistakes or misstatements that I may have made in summarizing their work.

The present issue of the Journal opens with an introduction to the basic concepts of evidence-based medicine (EBM), which is then transplanted from the general to the specific with a focus on forensic psychiatry. Our attention is drawn to the apparent similarity between these concepts and the standard set by both the American and Canadian supreme courts for the receiving of scientific evidence. The courts have held that the admissibility of scientific evidence (or the weight to be attached to it once admitted) depends, at a minimum, on its scientific validity. Courts determine scientific validity by applying the criteria set out in Daubert,¹ which include whether the theory or technique can and has been tested, has been subject to peer review, has a known error rate or the existence of standards, and has been generally accepted within the scientific community. (General scientific acceptance is arguably the weakest of the considerations. Once upon a time, it was generally accepted among the leading scientific luminaries of the day that the world was flat.) In a court of law, scientific validity is, of course, a necessary but not sufficient precondition in the determination of admissibility. Presumably, EBM will inform the court as to scientific validity.

A decision from the Supreme Court of Canada, R. v. Mohan,² deals with a related but quite different set of considerations with respect to expert testimony. The court typically hears from an expert (who, once scientific validity is established, relies on the scientifically valid evidence in forming an opinion), and it then determines whether the evidence of the expert is relevant (i.e., logically probative of an issue in dispute), and necessary (i.e., the trier of fact could not have determined the issue without the assistance of the evidence) despite its relevance. Ultimately, scientific validity, relevance, and necessity will be considered in determining admissibility (along with other criteria not of immediate interest to this discussion). Also of interest because of its apparent synchrony with EBM principles is the Supreme Court of Canada's decision in R. v. Abbey,³ which stands for the principle, inter alia, that an expert opinion is only as good as the factual foundation on which it is based.

In the medical arena clinicians are implored to consider the best available external evidence obtained from systematic empirical research in a “…conscientious, explicit, and judicious manner” (Ref. 4, p 71). This is EBM. At the risk of oversimplification, the message is: don’t employ a technique or procedure or prescribe a pill until you have satisfied yourself through the examination of empirical evidence that it has been demonstrated to work. This admonition appears rather trite to a lay observer who has presumed that EBM would have been alive and well and behind any of the pills one takes. I was heartened to read in other articles I was able to find⁵ that, in fact, EBM is nothing new and that good medicine has always, or at least from the middle of the 19th century, attempted to incorporate the best available evidence into approved practice. And while, at first blush, the whole notion appears eminently reasonable, some harsh words have come from authors who have called the term EBM “…an example of newspeak …” that “…would have delighted George Orwell” (Ref. 6, p 838). Others have criticized the EBM advocates for “…their arrogance, their jargon, and their penchant for denigrating others …the steps and recommendations of the evidence based medicine acolytes reek of obfuscation and platitudes” (Ref. 7, p 1171). Still others, as Graham Glancy and Michael Saini note,⁸ have criticized evidence-based practice (EBP) for being counterintuitive, process-driven, and too reductionistic and promoting a cookbook approach to medicine. We now have evidence-based everything. EB is now being attached in a bandwagon manner to every sort of endeavor. In addition to medical training, we now have clinical practice, research activity, health care management, purchasing, and policy making.⁹ Finally, while EBM may point the clinician down the right course, most of the time it will not be the right course for all patients all the time.¹⁰ Clinical judgment will be the final determiner, not some formula.

As well, from my lay perspective, all of this seems to assume that there is out there a singular agreed-upon objective lens through which scientific validity is assessed. Some questions came to mind. What is the most valid method of assessing the efficacy of a pill or a program? How do we assess the assessments? Who determines best evidence? Are randomized controlled trials necessarily the gold standard? Who will determine the sufficiency of the evidence? What is scientific? Do we need to go through an EBM assessment of the assessments? A search of MEDLINE on virtually any related topic will dump hundreds, if not thousands, of supposedly scientific papers onto the bewildered EBM researcher's lap. Technological advances inevitably make this sorting much less difficult, but it remains a daunting task that might provide more confusion than assistance. A major deficiency in the world of EBM is the well-known reality that negative results are extraordinarily difficult to publish and therefore difficult to find (other than in dissertation abstracts!). So, all of the research that might suggest that a particular theory, technique, or pill does not work is not readily accessible to the clinician doing EBM due-diligence. This reality is well known, but it does, however, strike right at the heart of the EBM approach. Available data tend to be positive results data. So, if you as a practitioner have a practice in place because its efficacy is consistent with clinical experience, studies that might show no benefit to that practice are probably going to be missed. Further, authors such as Levine and Fink¹¹ point out that in psychiatry a variety of imprecisions and complexities challenge the viability of EBM principles to such an extent that it is unworkable. This is unlike the world of pharmacology, where randomized controlled trials are relatively easy to contain and the effects easily isolated.

Criticisms notwithstanding, it is time to move on to a consideration of the excellent papers that I have been asked to read. Richard Rogers and Jill Johansson-Love¹² provide us with an examination of three published competency (to stand trial) measures: the MacArthur Competence Assessment Tool-Criminal Adjudication (MacCAT-CA), the Evaluation of Competency to Stand Trial-Revised (ECST-R), and the Competence Assessment for Standing Trial for Defendants With Mental Retardation (CAST-MR). Each of these instruments is carefully subjected to the Daubert analysis, and the strengths and weaknesses of each are discussed. All correlate highly with judicial decisions on fitness. However, the Daubert processor did not yield a clear winner. It would appear that, depending on the application, either of the instruments may be useful. The authors wisely recommend that the practitioner and the researcher, who have often been at odds, join forces. There are ups and downs (mostly downs) to subscribing rigidly to either philosophy. The researchers dismiss the practitioner's experience-based approach as softcore, inconsistent, and incomplete. The practitioners, as the authors note, “sometimes exaggerate the limitations of standardized measures while possibly overvaluing their own expertise” (Ref. 12, p 459).

The sound advice of the authors is that neither approach constitutes the full answer. In that regard, it was noted that one of the scales, the MacCAT-CA, had the most problems with its competency measures establishing accurate classification. The authors note: “Obviously, the group of hospitalized legally incompetent defendants should evidence clinically significant impairment, given their combined psychiatric and legal status” (Ref. 12, p 458). However, this was apparently not the case. As a judge sitting in a busy Mental Health Court where fitness to stand trial is sometimes litigated several times in a single day, I don’t find this result surprising in the least. Often, accused persons are found to be unfit to stand trial on very narrow grounds as a result of very tightly circumscribed thought disorder. In most every way the accused person may present and perform as normal; it is only when one pushes certain buttons that unfitness becomes glaringly apparent. So, an accused person may well score high on a standardized instrument, yet be found by the court to be unequivocally unfit. As well, fitness may fluctuate wildly between the scoring on a fitness measure and the accused person's appearance in court. This discrepancy brings home the point made by the authors that, while these instruments may be of tremendous assistance, the final assessment will be that of the practitioner who offers an opinion based on an array of information.

Actually, the final assessment is made by the court; that takes me to a point I’d like to make with respect to this particular enterprise (measurements of fitness/competency). The determination of fitness to stand trial is purely a legal decision, unlike, for example, what is the most effective treatment for anger and/or aggression. It is often, but not necessarily, informed by the input from mental health professionals such as psychiatrists or psychologists. Perhaps researchers would benefit from staying away from what is known in the legal arena as the ultimate issue in the design or objective of their instruments. There is a rule, unevenly adhered to, that an expert opinion should not approach the ultimate issue—that is, the central issue that the court has to decide. The concern is that the expert not usurp the function of the court. (This same problem obviously comes up in trials regarding criminal responsibility.) In an ideal legal world, the court would receive evidence with respect to the accused person's limitations, inabilities, and dysfunctions, and from all of that the court should proceed to make the legal determination as to whether this constitutes unfitness to stand trial. The legal determination of fitness to stand trial, of course, does not comport nicely with clinical reality, which would (as seen with the various instruments) demonstrate shades of fitness. Our law, as it is presently articulated, knows no shades: a defendant is either fit or unfit, no in-between. Is fitness then unmeasurable, in that the court must put the accused persons into discreet boxes marked fit and unfit? As a judge having to make these decisions, I can imagine that it would be most useful to hear how an accused person's abilities measure up against the norms and then, along with whatever other evidence is adduced, decide on a balance of probabilities whether the accused person is fit/competent. In fairness to the mental health experts, in my experience, they are not deliberate usurpers. It is the reticent or nervous courts that are for the most part quite content to receive as much assistance as possible on issues which, for the average judge, are rather exotic.

Another paper is that of Michael Saini, entitled “A Meta-Analysis of Psychological Treatment of Anger: Developing Guidelines for Evidence-Based Practice.”¹³ This very ambitious study consisted of a meta-analytic review of all relevant studies that had considered psychological approaches in the treatment of anger. (Of interest is that this rather mainstream search revealed 4,438 titles! One can imagine the filtering that a practitioner, engaging in EBM due-diligence, would have to employ to make any sense out of so much material.) The objective of this study is quite important, in that, as the author notes, there remains no consensus in the literature regarding the best way to treat or reduce anger and aggression. However, there are apparently many methods that have been successful. At this juncture, the evidence does not suggest a best practice. A comprehensive statistical analysis confirmed that while psychological treatments of anger were, as a group, generally successful, multicomponent therapy, an approach that is consistent with the focus on multitheoretical comprehensive multimodal approaches, appeared somewhat stronger. While this comprehensive approach seems to yield strong results, it is difficult to tease apart which of the many components are driving the effect. (An interesting finding noted by Glancy and Saini¹⁴ was that eight sessions is the optimal number beyond which attrition increases.) The authors note some variability of outcome depending upon sample setting and gender. The study appears to conclude that while psychological treatment of anger is generally effective, a therapy's effectiveness may vary as a function of gender, patient population, and practice setting. The data do not point to one-size-fits-all when it comes to the treatment of anger. The implication of this, so far as EBP is concerned, is that a very nuanced review of the literature is needed to find the treatment that is best suited to the practitioner's particular client.

Dr. Arturo Silva next provides us with a free ranging discussion of the emerging science of forensic neuropsychiatry. He alludes to a variety of considerations which might temper its utility in deciding matters before the courts.

Another paper in the line-up is by Frank Sirotich,¹⁵ who surveyed the literature to determine whether there is currently evidence to support the use of diversion initiatives to reduce recidivism or reduce incarceration among adults with serious mental illness within the criminal justice system. Diversion programs or, as they are typically known in the United States, mental health courts, are a response to the flooding of the criminal justice systems, both in Canada and the United States, over the past 15 years or so with mentally disordered accused persons. The provision of mental health care services in most western European and North American communities has witnessed a steady decline over this period. Beginning with the deinstitutionalization movement occurring in the later half of the 20th century, adequate mental health care services became increasingly scarce. Despite what was promised, the money saved with the closure of hospitals has typically not been reinvested in community treatment.

In some jurisdictions, mentally disordered accused persons entering the criminal justice system have increased at a rate in excess of 10 percent per year over the past dozen years.¹⁶ A criminalization of mental illness has occurred; a shifting of responsibility onto the criminal justice system for the provision of basic mental health care services.^17,18 In some jurisdictions mental health courts may respond to unmet constitutional guarantees (see, for example, Ref. 18).

Mental Health Courts and diversion programs have been a response to this dilemma. Despite sharing similar objectives, there are many models that claim the label of mental health court.¹⁹ Accordingly, when considering these courts, it is important to gain an appreciation of the scope of what is being referred to. Nevertheless, very generally speaking, mental health courts and diversion programs are all attempting a rehabilitative response to what would otherwise have been criminally sanctioned behavior.

While the nuts and bolts of mental health courts vary, integral to the functioning of a mental health court is a multidisciplinary team approach. Judges and lawyers are supplemented by any number of psychiatrists, psychologists, caseworkers, and social workers who collaborate on how the particular needs of the accused person can be met effectively.

Typically, participation in the diversion component of a mental health court is reserved for individuals with mental disorders charged with minor to moderately serious offenses. Nevertheless, certain courts also provide services that do not involve eligibility requirements.

In most mental health courts in the United States, eligible and consenting accused persons are given a choice: participate in a treatment program and have their criminal charges stayed, dropped, or reduced, or proceed in the regular stream. (While postarrest diversion programs are by far the most common, some jurisdictions are experimenting with prearrest diversion.) This approach has the benefit of not engaging the courts at all. The treatment program is strictly voluntary and the accused persons are most often able to opt out at any time. The approach in Canada is somewhat different. The primary focus is on assessing fitness to stand trial and providing treatment. The accused person's participation in this aspect of the court's operation is not voluntary. Thereafter, once found fit to stand trial, the accused person may elect to remain with the court for a bail hearing, participate in diversion, or resolve the matter with a guilty plea; it is the accused person's option.

Accused persons who elect to participate in the mental health court or diversion will typically be required to comply with an individually tailored treatment program designed by the mental health court team. (At the same time, some jurisdictions have a fixed program of a fixed duration in which all candidates enroll. It is the view of Sirotich¹⁵ that individually fashioned regimens adjusted to the individual's particular needs are more likely to be successful.) The benefit of a multidisciplinary team is that treatment can take a variety of forms and is not limited to medication, but can include psychological therapies, educational training, occupational training, housing and access to social services, and budgetary counseling, for example.

As Sirotich¹⁵ rightly notes, and others have commented for the past several years,²⁰ while these interventions have been eagerly embraced by the criminal justice system, empirical research testing their effectiveness has just started to appear over the past few years. The two principal questions sought to be answered by Sirotech are: “Do jail diversion initiatives for adults with serious mental illness reduce criminal recidivism?” and “Do such initiatives reduce time spent in custody by these adults?” (Ref. 15, p 464).

Measures of effectiveness in the present study were limited to recidivism and time in custody. Twenty-five articles that met certain stringent criteria were located through a search. For example, failure to include a comparison group in the design resulted in exclusion of the study. Prebooking (or prearrest) and postbooking (postarrest) diversion programs were considered. The combined results of the studies considered were ambiguous but, perhaps, encouraging. The studies considered were seen as demonstrating little evidence of the effectiveness of jail diversion in reducing recidivism rates. However, the amount of time spent in jail was less for the mental health court group. The author concludes that further study is needed.

As well, the success of a program should be measured in many ways other than recidivism. It is important to look carefully at the sorts of offenses for which accused persons are being rearrested, as well as the frequency. One might predict, for example, that participants in diversion will have higher recidivism rates and more frequent mental health contacts given that they are now on the radar screen.

An inherent problem in assessing the efficacy of most diversion programs or mental health courts is that there are no singular programs. In the United States, there are well over 100 programs, no two of which are the same. The diversity of programs is not necessarily a bad thing, in that they typically arise in response to particular local needs and employ locally available resources. Also, in most jurisdictions, when the accused person agrees to participate in diversion, a specific program is designed for that individual as a function of his particular needs. The programs may engage a wide and differing range of services and the duration of the program will vary from individual to individual as a function of the accused person's rate of progress and stabilization. So, beyond the broad form of the process, there is not a consistent therapy or intervention to assess.

Nevertheless, other studies, no doubt employing less rigorous experimental designs, examining other aspects of mental health courts and diversion, although methodologies are variable,²¹ have provided broader information as to the efficacy of mental health courts. (An excellent article suggesting an approach to the collection of mental health court data has been produced by the Bureau of Justice Assistance.²²) There are now studies that support the previously intuitive projection that mental health courts do indeed reduce recidivism rates.²³ Studies are now showing that participation in mental health court programs is associated with longer time without any new criminal charges, or charges for violent crimes.²⁴ In addition to reducing the probability of future arrests, data are now confirming that those who complete their mental health court programs do better than those who do not.²⁵ Other reports indicate that mental health courts improve access to care,²⁶ save the taxpayers money by keeping mentally ill individuals out of prison, reduce drug abuse, improve overall levels of functioning,²⁷ and should no longer be funded on a pilot project basis.²⁸

It is true, as Sirotich¹⁵ indicates, that more study is needed. But, what to do with that hiatus between the development of a procedure or program and the collecting of the data to show whether the intervention is effective or what is the best evidence-based practice? Or, what to do when the process is not quantifiable? All three of the studies in this issue of the Journal in which the authors have looked at fairly mainstream enterprises within the world of forensic psychiatry have failed to yield unequivocal answers as to what is the best practice. It may be that EBM principles, as applied to forensic psychiatry or to psychiatry in general, will at best be able to identify programs, therapies, and interventions that are effective for some people, some of the time, in certain circumstances. Having recognized some significant challenges, it is nevertheless my view that every effort must continue in assessing new therapies or interventions to ensure that they do indeed have some beneficial effect on accused persons.

In addition, it is essential to the success of the EBM approach that the scientific community insist that its journals and publications of all sorts publish all methodologically sound studies regardless of whether the results are positive, negative, or show no effects at all. If the scientific community persists with the tradition of burying studies that show no effects, the EBM approach is of extremely limited value, or worse. We must be sure that these therapies not only improve the accused person's situation, but that they do no harm. No-effect and negative-effect studies are as important to know about as are those with positive results and must be equally accessible to practitioners. This requirement must be endorsed as a basic essential foundation if the EBM approach is to be of any use.

• American Academy of Psychiatry and the Law