Abstract
In this second part of our study, we examined the extent of agreement between treating practitioners and expert witnesses on psychiatric diagnoses in evidence presented in criminal proceedings. We found good agreement on diagnoses of acquired brain injury, schizophrenia-spectrum psychoses, depressive disorders, intellectual disability, substance abuse, and personality disorders; fair agreement on substance-induced psychotic disorder; and poor agreement on the presence of anxiety disorders. A proportion of defendants with diagnosis by experts of substance-induced psychotic disorder also had a diagnosis of schizophrenia-spectrum psychosis by treating practitioners. Treating practitioners and experts engaged by the prosecution rarely made the diagnosis of post-traumatic stress disorder. Overall, there was moderate agreement between experts and treating practitioners on the principal Axis I disorder, and the evidence for psychiatric diagnoses presented by treating practitioners in criminal cases was found to be generally reliable.
Treating practitioners are often in a position to provide assistance to courts in cases involving patients under their care, because of their familiarity with the patients’ psychiatric conditions and circumstances. However, a conflict in ethics could arise from an understandable desire to help their patients by providing favorable evidence, a wish to avoid harming their patients or the therapeutic relationship by disclosing prejudicial information, and their duty to maintain confidentiality. The potential for such conflict has led some commentators to conclude that asking treating practitioners to give evidence in court cases has the potential to compromise the quality of mental health care, and some have recommended that treating practitioners avoid giving evidence about their patients.1
It has even been argued that the roles of treating practitioner and expert witness are irreconcilable because the ethics-based duties arising from providing treatment compromise the nature of the information that a treating practitioner can reveal,2 and therefore treating practitioners should be barred from providing expert evidence.3 Recommendations against the presentation of evidence by treating practitioners can be found in the codes of ethics of organizations such as the American Academy of Psychiatry and the Law,4 the American Psychological Society,5 and the Royal Australian and New Zealand College of Psychiatrists.6 However, not all professional bodies agree: the American Psychiatric Association7 and the Academy of Medical Royal Colleges8 stress the need for objectivity and professionalism when giving evidence, without making any specific recommendations for treating practitioners.
Despite long-standing reservations about the reliability of evidence given by treating practitioners,9 few empirical studies have been conducted to determine whether such evidence is biased. It might be assumed that a treating practitioner would attempt to assist his patient and that the evidence would be closer to that of an expert engaged by the defense than one engaged by the prosecution. However, in a study of written evidence provided in civil matters, experts and treating practitioners were found to be just as likely to agree on psychiatric diagnoses as were experts engaged by the opposing sides.10 Any study of bias among treating practitioners must also consider the possibility of bias in the experts’ evidence to which the treating practitioners’ evidence is compared.11–15 However, in Part 1 of this study16 and in a related study of evidence about the availability of mental illness defenses and competence to stand trial,17 we found little evidence of bias among experts in criminal matters.
In this part of the study, we examined the level of agreement on psychiatric diagnoses between treating practitioners and expert witnesses in serious criminal cases. We did not consider opinions about the availability of a mental illness defense or competence to stand trial, because in this series, treating practitioners rarely provided opinions on legal matters. The written reports of treating practitioners were mostly prepared at the request of the defendants’ legal representatives, who then served the reports on the prosecution, although a few of the reports were written in response to a judicial request for a report about the condition of a defendant held in custody or in a hospital.
Initially, we examined the levels of agreement between treating practitioners and experts engaged by the defense or prosecution on the major diagnostic categories and the principal Axis I diagnosis. If treating practitioners were biased in favor of their patients, they might be expected to diagnose more serious conditions to assist their patients in court.
We then tested two hypotheses:
First, that there is more likely to be agreement on the principal diagnosis in pairs of reports in which one of the reports was written by a defense expert and the other by a treating practitioner and that agreement is less likely in pairs of reports written by a prosecution expert and a treating practitioner. An association between agreement on the principal diagnosis and report pairs that included a defense expert's report would suggest that the diagnoses of treating practitioners tend to favor the defendant.
Second, that there is more likely to be agreement on the principal diagnosis made in pairs of reports by mental health professionals from the same profession (psychiatrist or psychologist) and that agreement is less likely in pairs of reports written by mental health professionals from different professions. An association between the professions of the experts and agreement on the diagnosis would suggest that the professional's training and experience influence their diagnoses.
We also examined the possibility that there is a difference in the written evidence of treating practitioners and the clinical diagnoses, by examining the agreement in pairs of reports that included an expert report and a treating practitioner's written report and an expert report and the clinical diagnosis taken from prison medical records. This analysis was conducted to examine the possibility that the clinical diagnosis that was not provided to the court would be less likely to agree with the expert's diagnoses than the diagnosis provided in the treating practitioner's written reports.
Finally, we examined the possible associations of age, sex, marital and employment status, the history of convictions, and the nature of the charges with agreement on the principal psychiatric diagnosis, because of the possibility that agreement on the psychiatric diagnosis could be associated with these demographic or criminologic variables.
Methods
The Sample of Reports
The sample of experts’ reports is described in Part 1.16 In summary, the reports came from 110 consecutive cases in which the prosecution obtained a written report, from which there were 270 reports written by experts who had no role in treatment and 34 reports by treating psychiatrists or psychologists that were written after the offense but before the trial. We were also able to obtain a treating psychiatrist's clinical diagnosis from the New South Wales Justice Health medical records for 28 additional defendants who had received treatment while in custody. Hence, in this study we examined a series of cases in which there was a report by at least one type of expert (prosecution or defense) and a diagnosis made by a treating practitioner. The treating practitioners’ diagnoses included those made in 34 written court reports and the clinical diagnoses of 28 other defendants by practitioners providing treatment in prison.
Permission to perform the study was obtained from the NSW Justice Health Research and Ethics Committee and the NSW Director of Public Prosecutions.
Data Collected
We collected the following data:
whether the evidence was provided by an expert or a treating practitioner;
whether the expert had been engaged by the defense or the prosecution;
whether a treating practitioner's diagnosis was taken from medical records or from written reports served in court;
whether the expert was a psychiatrist or a psychologist;
the sex of the defendant and the age, marital status, and employment status at the time of the offense;
the most serious current charge and history of convictions;
and the psychiatric diagnoses.
The diagnosis made by experts and in treating practitioners’ reports were collected in a reliable way by M.L. and O.N. or G.E., as described in Part I.16 Clinical diagnoses were extracted from the prison medical records by G.E., who confirmed the diagnosis with the inmate's treating practitioner when necessary.
Coding of the Diagnoses
Australian psychiatrists generally adhere to the DSM diagnostic system, and the diagnoses made by the expert witnesses in this study were generally consistent with Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV).18 The diagnoses were coded according to categories that were consistent with DSM-IV chapter headings, and the methods described in earlier research were used in the analysis of agreement.10 Hence, schizophrenia, schizophreniform disorder, schizoaffective disorder, delusional disorder, and psychosis not otherwise specified, were grouped together in the analysis of agreement as schizophrenia-spectrum disorders. Major depressive disorder, dysthymia, and adjustment disorder with depressed mood were grouped together as depressive disorders. PTSD was included with other anxiety disorders, including obsessive-compulsive and panic disorders. Non-DSM categories included acquired brain injury, intellectual disability, and psychosis. The category of acquired brain injury included traumatic brain injury, alcohol-related brain damage, and dementia. Intellectual disability was coded if the expert diagnosed developmental disability of any severity including borderline low intelligence. A further analysis was conducted of diagnoses in a broader category of psychotic disorder that included cases of schizophrenia-spectrum psychosis, substance-induced psychotic disorder (SIPD), psychotic depression, and mania, because all of these conditions can be associated with a reduction in criminal responsibility.
Statistical Analysis
The methods used to assess agreement between treating practitioners and experts were similar to those described in earlier studies of the reliability of psychiatric evidence in civil matters.10,16
The overall level of agreement between experts and treating practitioners about the broad DSM categories, acquired brain injury, intellectual disability, psychosis, and the principal Axis I diagnosis was assessed with the kappa statistic, with agreement defined as poor, <0.2; fair, 0.2 to 0.4; moderate, 0.4 to 0.6; good, 0.6 to 0.8; and very good, 0.8 to 1.0.19 In this analysis, no distinction was made between experts engaged by defense or by the prosecution and hence the diagnoses of all experts were compared with those of all the treating practitioners. In contrast to our earlier10 and related studies,16 all the report pairs were statistically independent, because we did not examine the agreement between pairs of experts in cases with three or more reports, and there were no pairs of treating practitioners. Fisher's exact test was used to examine a possible association between expert and treating practitioner status and the diagnoses for those disorders that did not have good or very good agreement.
Univariate and multivariate generalized estimating equations (GEEs) were used to examine the associations of the role of expert (defense or prosecution), the profession of the mental health practitioners (psychiatrist or psychologist), and the type of treating practitioner (written court report or clinical diagnosis) with agreement on the binary dependent variable of principal Axis I diagnosis. GEE allow an examination of clustered data, multiple report pairs generated in cases where there was a single treating practitioner's report and the reports of two or more experts. The principal Axis I diagnosis was defined by the hierarchy of acquired brain injury, schizophrenia-spectrum psychosis, substance-induced psychotic disorder (SIPD), other psychoses (including mania), depressive disorders, anxiety disorders, and other Axis I disorders except substance abuse and dependence. Substance dependence and abuse disorders were not considered in the overall measure of agreement, because these disorders were inconsistently reported in the body and conclusions of many reports. Personality disorders were not included in the analysis, although there were no cases in which there was both disagreement on an Axis I disorder and agreement on the presence of personality disorder. The independent variables selected addressed the first and second hypotheses, including whether the pair of diagnoses included a diagnosis made by a prosecution expert and whether the report writers were in the same profession. The inclusion of a clinical diagnosis from the prison medical record, and the other independent demographic and criminologic variables, including age, sex, marital status, employment status, and history of criminal convictions and whether the charge was murder or attempted murder, were also examined in the GEE analysis.
A statistical power analysis estimated that 108 report pairs were necessary for the study to have an 80 percent chance of finding a true association (at p < .05) between factors that were present in half the report pairs and that were twice as likely to be present in report pairs in which there was agreement rather than disagreement between the experts and treating practitioners about the principal Axis I diagnoses.20 All the statistics were performed with SPSS for Windows, version 15.0.
Results
The 62 cases in which there was diagnosis by a treating practitioner included 5 with one expert report, 34 with two expert reports, 21 with three expert reports, and 2 with four expert reports. Hence, there were 144 pairs of reports that included a report by a treating practitioner and an expert: 5 + (34 × 2) + (21 × 3) + (2 × 4).
The level of agreement between expert witnesses and treating practitioners was rated as good for acquired brain injury (κ = 0.754), schizophrenia-spectrum psychoses (κ = 0.736), any psychosis (κ = 0.699), depressive disorders (κ = 0.662), intellectual disability (κ = 1.0), substance abuse (κ = 0.607), and personality disorders (κ = 1.0). There was fair agreement on SIPD (κ = 0.308) and poor agreement on anxiety disorders (κ = 0.114). Agreement on the specific principal Axis I diagnosis was moderate (κ = 0.404; Table 1).
Agreement About the Psychiatric Diagnosis Between Treating Practitioners and Experts*
Agreement on the diagnosis of SIPD was fair. In 8 of the 10 defendants in whom an expert diagnosed SIPD, the treating practitioner diagnosed schizophrenia-spectrum disorder. In this sample, experts engaged by the prosecution were more likely to make the diagnosis of SIPD (7/57 reports) than were experts engaged by the defense (1/87 reports, Fisher's exact test, two-tailed p = .007).
The kappa statistic for anxiety disorders was also low. Post-traumatic stress disorder (PTSD) was the most commonly diagnosed anxiety disorder, but it was not reliably diagnosed. One treating practitioner, six of the 87 defendants’ experts, and none of the 57 prosecution experts made the diagnosis of PTSD (Fisher's exact test, two-tailed p = .003).
Multivariate GEEs showed that the inclusion of a report by a defense expert rather than a prosecution expert did not result in a higher probability of agreement on the principal diagnosis (Table 2). However, treating practitioners and experts were more likely to agree if both reports were written by either psychiatrists or psychologists, compared with pairs of reports in which one was written by a psychiatrist and one by a psychologist.
Generalized Estimating Equation Analysis of Factors Associated With Agreement Between Experts and Treating Practitioners About the Principal Axis I Diagnosis
Multivariate GEEs showed that agreement on the principal diagnoses was more likely if the treating practitioner's diagnosis was provided in a written court report rather than obtained from the prison medical record. Univariate GEEs indicated an association between prior convictions and agreement on the principal diagnosis and a lower probability of agreement in homicide cases when compared with other offenses. Multivariate GEEs showed that the association between prior convictions and agreement on the principal diagnosis was independent of other factors. However, the presence of a homicide charge was not significantly associated with agreement or nonagreement in the multivariate analysis. Finally, multivariate GEEs also showed that agreement on the diagnosis between treating practitioners and experts was associated with the variables of single marital status, older age, and employment status at the time of the offense and a history of prior convictions.
Discussion
The level of agreement between experts and treating practitioners was good or very good for acquired brain injury and intellectual disability and for schizophrenia-spectrum, depressive, substance abuse, and personality disorders. Agreement on the specific principal Axis I disorder was moderate. Agreement on SIPD disorders was fair, and agreement on anxiety disorders was poor.
The diagnosis of schizophrenia by treating practitioners in some of the cases in which the expert made the diagnosis of SIPD contributed to the lower kappa value for SIPD when compared with other psychotic disorders. This result should be interpreted cautiously because the level of agreement on the diagnosis of SIPD between defense and prosecution experts was good in the larger sample of cases described in part 1 of this study.16 This result could be due to selection bias arising from the ongoing treatment of defendants with underlying psychotic illness who were more likely to have treating practitioners while in prison and hence a clinical diagnosis in their medical records. In contrast, defendants with a diagnosis of SIPD who did not have an underlying psychosis might not have had contact with prison medical services and would be less likely to have a treating practitioner. Moreover, a cross-sectional assessment by an expert concerned with the defendant's state of mind at the time of the offense could highlight the effect of recent substance use, whereas the defendant's treating practitioner could have reached the diagnosis of schizophrenia after a longer period of observation in the absence of substance use.
The other diagnostic group with a lower level of agreement was anxiety disorders—in particular, PTSD. In no case in which a defense expert diagnosed PTSD did the prosecution expert agree with the diagnosis. In the only case in which a treating practitioner made a diagnosis of PTSD, neither the defense nor the prosecution expert agreed with the diagnosis. SIPD and PTSD are unusual in that they require the clinician to decide on the cause of the symptoms to make the diagnosis. Hence, disagreement on the cause of the disorder rather than on the symptoms may have contributed to the disagreement on the diagnosis.
There was no evidence of an association between the roles of the experts (prosecution or defense) in the report pairs and agreement on the principal psychiatric diagnosis, indicating that the diagnoses of treating practitioners were not closer to those of defense experts.
The finding of a higher probability of agreement between experts and treating practitioners from the same profession should be interpreted with caution because of the relatively small proportion of reports prepared by psychologists and the absence of significant disagreement on the basis of profession in our related study.16 However, the finding is not altogether surprising, because there have been several legal challenges to the psychiatric diagnoses made by psychologists in NSW, which may have made some psychologists circumspect in expressing opinions about the presence of psychotic illness, the most common diagnostic category in the study.
We found that the diagnoses in the treating practitioners’ written reports were more likely to agree with those of experts than were the clinical diagnoses found in the medical records, perhaps because the experts were often aware of the treating practitioners’ written reports and may have accepted the treating practitioners’ diagnoses. However, in our view, a more likely reason is that the psychiatric diagnoses in the prison medical records were extracted some time after the trial, whereas the written reports by treating practitioners were prepared between the offense and the trial, around the same time as those of the experts. This was particularly relevant to the defendants with a diagnosis of SIPD made at the time of the trial, who were later given a clinical diagnosis of schizophrenia because of persistent or recurrent psychotic symptoms while in prison.
The findings related to age, employment status, and marital status should also be interpreted with caution, because we had not formed any hypotheses relating to these variables and did not perform a statistical correction for the number of factors that we examined. Moreover, only a relatively small proportion of the defendants were employed or married. The association between agreement and previous convictions may have been due to the availability of documents from previous matters in some cases.
The main limitation of this study stems from the lower than expected number of pairs of reports in which the experts disagreed about the principal diagnosis, which reduced the statistical power. The lack of statistical power raises the possibility of a type II error. Hence, a larger study could have found statistically significant associations between some factors included in this study and agreement between treating practitioners and experts. Another limitation was the observational nature of the study, as the expert witnesses may have relied on the treating practitioner's diagnoses, increasing the probability of agreement. The study also included cases in which there was more than one report for each side, which could have been obtained because of differences in the conclusions of previous reports. Additional reports obtained for this reason would be expected to reduce the overall agreement.
The cases examined in this study were all serious criminal matters, and most of the defendants were found to have a major psychiatric disorder. Hence, the findings may not apply to less serious offenses or to defendants with less serious mental disorders. In addition, the nature of the study did not allow us to examine ethics-related concerns regarding treating practitioners’ evidence—for example, whether adequate consent was obtained to disclose information or whether the therapeutic relationship was harmed when the treating practitioners provided written reports.
Despite these limitations, it appears that in serious criminal matters, the psychiatric diagnoses made by treating practitioners are similar to those made by expert witnesses, a finding that should improve confidence in both forms of opinion. We do not go so far as to recommend that treating practitioners provide evidence in court cases about their patients, because there are strong arguments regarding the ethics of encouraging or compelling treating practitioners to give evidence.1–3 Moreover, our study shows that in serious criminal matters, the opinions of treating practitioners are generally similar to those of experts. Hence, the results support the use of an expert assessor, but equally do not support the exclusion of evidence by treating practitioners on the grounds that it may not be reliable.
Conclusions
We found generally good agreement on psychiatric diagnoses in written evidence provided by treating practitioners in serious criminal matters. The findings of this study suggest that concerns about bias arising from the nature of a treating practitioner's relationship with a patient may be overstated, at least in relation to criminal proceedings. However, the level of agreement on the diagnoses of both PTSD and SIPD was lower than for other diagnoses, and experts should take particular care to review corroborative information and to consider all of the diagnostic criteria when making the diagnoses of PTSD or SIPD.
Acknowledgments
We thank Nicholas Cowdrey, QC, New South Wales Director of Public Prosecutions, for permission to conduct the study; Craig Hyland, Office of the Director of Public Prosecutions, for assistance in collating the reports; and Peter Arnold for invaluable assistance in preparing the manuscript.
Footnotes
-
Disclosures of financial or other conflicts of interest: None.
- American Academy of Psychiatry and the Law
