Abstract
The authors present an important two-part study as they strive to provide an empirical analysis of psychiatric diagnoses in criminal case reports in Australia. In the first part, they compare the level of agreement or correlation of diagnoses between pairs of experts who prepared reports for either the prosecution or defense with other reports prepared for the same and opposing sides and by profession (i.e., psychiatrists and/or psychologists). In the second part, they compare the level of agreement or correlation between experts retained by either the prosecution or defense and treating practitioners. Psychiatric diagnoses are fundamental requirements that may affect the adjudication of criminal and civil cases. Both parts of the study focus on criminal cases and are very exciting in that they review not only the correlation of agreements in these areas but also address indirectly the concept of the so-called hired gun. The development of specialized expertise in the evaluation and assessment of defendants by designated opinion or expert witnesses has progressed over time. The nexus between psychiatry and the law (i.e., forensic psychiatry) has included the presentation of psychiatric diagnosis to the courts and the necessity for the expert or treating practitioner to address legal questions raised by the court. This study makes important steps in the direction of examining and analyzing the role of psychiatric diagnosis according to the responsibilities of the evaluator (i.e., as independent examiner or treating practitioner), as well as the possible influence of professional training and experience on differences in diagnoses between two evaluators. It is anticipated that there will be further work in these areas to address not only diagnoses but forensic recommendations and opinions.
In a two-part study, Nielssen et al.1 and Large et al.2 took on the challenge of presenting empirical data and information on several aspects of psychiatric diagnoses by expert witnesses and treating clinicians presented in criminal cases in Australia. In the first study, they looked at variables including the agreement on diagnosis between experts retained by the same or opposing sides (i.e., prosecution and defense) and the agreement on diagnoses between professions (i.e., psychiatrists and psychologists). In the second study, they examined the same variables but determined the level of agreement on psychiatric diagnoses by treating psychiatrists and/or psychologists and forensic examiners.
These articles are very welcome, as they include a review of the literature regarding expert witnesses acting as so-called hired guns—that is, the possibility that their opinions are biased by those who retain them.3–5 These studies by the authors are very helpful in attempting to address multiple issues encountered in forensic psychiatry and psychology. Foremost among them is the perception of the hired gun. Gutheil defined the hired gun as “an expert witness who sells testimony instead of time” (Ref. 6, p 7). He went on to state that psychiatrists, including forensic psychiatrists, sell their time, but the hired gun “goes beyond” selling time to “demonstrate corrupt willingness to offer for money the testimony that the retaining attorney desires, regardless of its clinical or empirical validity” (Ref. 6, p 7). Inter-rater reliability for psychiatric diagnoses has been noted as good to excellent for most Axis I conditions defined in Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) and International Classification of Diseases, 10th edition (ICD-10), provided clinicians use the necessary diagnostic criteria.7
The authors compared their analyses of information gathered from criminal case records to those in previously published works regarding civil cases and found striking differences.8,9 I am not at all surprised that there are differences in their findings in criminal versus civil matters, for reasons that the authors themselves provide. In the criminal matters reviewed in the study, experts who were retained by the prosecution often came from a panel of forensic psychiatrists, the majority of whom were in clinical practice and had prepared reports for the defense in other cases. Experts retained by the defense were often listed on the prosecution panel of experts or were known for their experience and reputations. The authors report that in civil matters the body of experts may include practitioners who have retired and are devoting their professional time to participation in forensic civil matters and/or others who have not been listed on such a panel. The authors also note that in many civil cases, experts are not required to provide oral testimony and experience the rigors of cross-examination.
In my experience, in the United States the range of available experts for civil matters can be quite extensive. They can include practitioners who have no forensic experience whatsoever and may or may not be known for their experience and reputations. Also, my experience has confirmed that in many of these cases, the proposed experts may have a very limited understanding of the actual parameters of civil matters, including not only DSM-IV diagnoses but also the questions that are applicable to the legal proceedings in medical malpractice, testamentary capacity, child custody, and other civil matters. In criminal cases, I have also witnessed the participation of experts who have a particular area of expertise but very little understanding of the basic standards involved in forensic work and the law and therefore may make statements in their written reports or during oral testimony that are inconsistent with the legal parameters or questions in a given case. For example, I have knowledge of experts who have testified to diminished capacity in jurisdictions where diminished capacity is not a legal option. I also know of cases in which experts have testified regarding competence to stand trial or criminal responsibility in which they have interviewed only the defendant and have not reviewed other important documents or collateral information regarding the defendant's history and current circumstances in a treatment facility or jail.
In their work, the authors strived to apply empirical data to this complex process but acknowledged several limitations in the sample size and correlation of data. They demonstrated that the correlations between experts in criminal matters, whether retained by the prosecution or the defense, were higher than perhaps anticipated in several categories and poor in at least one category, which I will describe further. They also reported their findings that the concordance between professions was substantial and that the concordance between treating clinicians and expert evaluators was good.
The Correlation Between Experts
The authors examined 110 criminal cases between 2005 and 2007. Two hundred seventy reports were produced, 226 by psychiatrists and 44 by psychologists. Of the 270 reports, 148 had been written by experts retained by the defense and 122 by experts retained by the prosecution. In their efforts to present empirical rather than anecdotal evidence, the authors cited appropriate references on anecdotal evidence and made reference to the Daubert standard.10 The Daubert standard, established by the United States Supreme Court in 1993, has as its base that it is the trial judge's responsibility to evaluate the scientific validity and admissibility of expert testimony under the Federal Rules of Evidence. In the first part of the study,1 the authors set forth two hypotheses regarding the reliability of evidence for psychiatric diagnoses by experts:
That there is an association between agreement by experts on the same side and less likely agreement between experts on opposing sides.
That there is an association between pairs of evaluators from the same professions. (In other words, it would be more likely that psychiatrists would agree with other psychiatrists and psychologists would agree with other psychologists.)
The authors also looked at different demographic factors including age, sex, marital status, and employment status and criminologic factors including history of criminal convictions and the charges in the current case, to determine correlations of these factors with agreement between evaluators. I did not find the consideration of demographics or criminal charges as factors particularly useful in the results as they were presented.
The authors report their purpose to be assessment of the extent of agreement on psychiatric diagnosis by experts in serious criminal matters, and their findings were as follows:
Good or very good level of agreement between experts regarding substance-induced psychiatric disorders (SIPD) and intellectual disability (defined further in the paper as mental disability or borderline intellectual functioning).
Good agreement between experts on diagnoses of acquired brain injury, schizophrenia-spectrum psychosis, personality disorders, and substance misuse.
Moderate agreement regarding depressive disorders and personality disorders.
Poor agreement regarding anxiety disorders, particularly post-traumatic stress disorder (PTSD).
The authors also report that there was moderate agreement regarding the principal, or primary, Axis I diagnoses by experts retained by the same side (i.e., experts hired by either the prosecution or defense) and moderate agreement between experts hired by opposing sides in a criminal matter.
The Correlation Between Experts and Treating Practitioners
In their second paper,2 the authors examined the level of agreement between experts and treating practitioners and tested the following two hypotheses:
That there is an association between agreements on the principal diagnoses in pairs of reports in which one report was written by a defense expert and the other a treating psychiatrist, and less likely agreement in pairs of reports by a prosecution expert and treating psychiatrist. This result suggests that the diagnoses of treating practitioners would favor the defendant.
That there is an association between agreement on the principal diagnosis in pairs of reports by mental health professionals from the same profession (i.e., psychiatrist or psychologist) and less agreement in pairs of reports by those in different professions. This result suggests that professional training and experience influences diagnoses.
The authors used the same database of 110 criminal cases between 2005 and 2007. Two hundred seventy reports were written by experts who had no role in the treatment of the individual defendants and an additional 34 reports were provided by a treating psychiatrist or psychologist and were written after the offense but before the trial. They also reviewed the medical records of 28 defendants who had received treatment while in custody. The diagnoses made by the psychiatrists were generally consistent with the DSM-IV definitions and included categories for schizophrenia-spectrum, depressive, and anxiety disorders, with PTSD regarded as an anxiety disorder. A category of acquired brain injury included traumatic brain injury, alcohol-related brain damage, and the dementias, and an intellectual disability category included defendants with diagnoses of mental disability or borderline low intelligence. Finally, substance-induced psychotic disorder (SIPD) was included; however, substance dependence and abuse disorders were not included in the review. As in the first study, the authors calculated their results by using the kappa statistic and univariate and multivariate generalized estimating equations (GEEs). They applied the analysis to 62 cases in which there was a diagnosis rendered by a treating practitioner paired with from one to four expert reports and found 144 pairs of reports.
The authors found that the level of agreement between expert witnesses and treating practitioners was as follows:
Good or very good for acquired brain injury, schizophrenia-spectrum psychosis, any psychosis, depressive disorders, intellectual disability, substance abuse, and personality disorders.
Fair between experts and treating practitioners for SIPD. With regard to cases in which an expert diagnosed SIPD and a treating practitioner diagnosed schizophrenia-spectrum disorder, experts retained by the prosecution were more likely to make a diagnosis of SIPD than were experts engaged by the defense. The authors opined that the level of agreement on the diagnosis of schizophrenia by treating practitioners in cases in which an expert made a diagnosis of SIPD was lower than that for other disorders because of a selection bias arising from the continuing treatment while in prison of defendants with underlying psychotic illness versus defendants with an SIPD diagnosis who did not have an underlying psychosis and may not have received treatment while in custody.
Poor agreement, the lowest level of agreement, for anxiety disorders. One of the most striking results of the review was that PTSD, which is the most commonly diagnosed anxiety disorder, was not reliably diagnosed. Only 1 report by a treating practitioner, 6 of 87 reports by defense experts, and none of 57 reports by experts retained by the prosecution included a diagnosis of PTSD. The authors noted that in no case in which a defense expert diagnosed PTSD did a prosecution expert agree with the diagnosis; and in the only case in which a treating practitioner made a diagnosis of PTSD, neither the defense nor the prosecution expert agreed. The authors offered that both SIPD and PTSD are unusual, as they require the clinician to decide on the cause of the symptoms to make the diagnosis.
With regard to the level of agreement by profession, there was a higher probability of agreement on the principal diagnosis between an expert and the treating practitioner if both reports were written by either psychiatrists or psychologists compared with pairs of reports written by a psychiatrist and a psychologist.
In my experience as a practicing forensic psychiatrist, the difficulties in both criminal and civil cases with diagnoses of SIPD and PTSD arise, in part, because both diagnoses rely heavily on self-reporting by the defendant. In many cases, corroborating external data supporting the diagnosis of either of these conditions are limited.
The authors also reviewed 34 written reports by treating practitioners and 28 clinical diagnoses recorded in the medical records by treating practitioners. They found that the level of agreement when comparing experts' reports with those written by treating practitioners was greater than when comparing them with the clinical diagnoses in the medical records. They opined that the difference in correlation may have been caused by the experts' awareness of the diagnoses in the treating practitioners' written reports and by their acceptance of those diagnoses. They also offered that the preparation of the experts' and treating practitioners' reports in roughly the same time frame (i.e., between the time of the offense and the trial) may have caused the higher level of agreement on the principal diagnoses.
The authors concluded that the major limitation of the study was the lower than expected number of pairs of reports in which the experts disagreed about the principal diagnosis. They asserted that a larger study would find statistically significant associations between some of the factors included in their study and agreement between treating practitioners and experts. Although the study led to the conclusion that the psychiatric diagnoses made by treating practitioners are similar to those made by expert witnesses, the authors opined that compelling or encouraging treating practitioners to provide evidence about their patients in court cases is ethically questionable. The study supported the use of experts, but did not support the exclusion of evidence by treating practitioners on the grounds that it might not be reliable. The authors further concluded that concerns about bias arising from the nature of the treating practitioner's relationship with the defendant could be overstated in criminal proceedings. They were also self-critical in suggesting that caution be used in interpreting the results that showed agreement between experts and treating practitioners of the same profession, because of the low number of reports prepared by psychologists and the absence of significant disagreement on the basis of profession in their second study. In my view, the authors demonstrated that forensically trained psychiatrists agree more frequently than not on diagnosis, an additional important finding. This finding is consistent with my experience in forensic work.
The review of empirical data on the levels of agreement between experts and between experts and treating practitioners is informative, but the authors appropriately cite limitations and cautions. Although the analyses focused on levels of agreement on diagnoses, a very useful study would be the level of agreement on the recommendations suggested by experts and treating practitioners in response to the forensic or legal questions raised by the court, such as opinions regarding competence to stand trial and criminal responsibility. I applaud the effort by the authors and look forward to future articles on these important forensic matters.
Footnotes
-
Disclosures of financial or other potential conflicts of interest: None.
- American Academy of Psychiatry and the Law
