Abstract
This article describes how the establishment and existence of a forensic psychiatry fellowship program was associated with improvements in general psychiatry residents’ scores on the Psychiatry Resident In-training Examination (PRITE). Four consecutive years of general psychiatry residents’ PRITE scores spanning 2 years before and 2 years after implementation of the forensic fellowship program at our institution were compared. Mixed-model statistical analyses accounting for repeated measurements of individual residents across the periods indicated statistically significant improvement in forensic content scores and several other subspecialty areas in which our institution offers educational fellowship programs. External indicators of program outcomes such as standardized examination scores may provide a useful indication of the effects that an educational fellowship program can have on general psychiatry education.
Forensic psychiatry is a relatively new and rapidly growing subspecialty in medicine, both in the United States and internationally.1 The Accreditation Council for Graduate Medical Education (ACGME) began accrediting forensic psychiatry training programs in 1996,1 and currently there are approximately 40 such accredited programs in the United States.2 In 2004, the American College of Psychiatrists added forensic psychiatry as a specific domain to be assessed on the Psychiatry Resident In-training Examination (PRITE).
Curriculum and educational topics in forensic psychiatry fellowship training have been described recently in the literature.3–6 In brief, forensic psychiatry fellowships provide trainees with the skills necessary to interact with the legal system and to provide expert testimony in the civil and criminal arenas. In addition, trainees gain clinical experience while treating patients in correctional facilities.7 Several recent articles have also discussed forensic psychiatry topics that would be of most benefit if incorporated in curricula for general psychiatry training programs.8,9 Novel case-based instructional strategies that can be incorporated by educators who teach forensic topics to general psychiatry residents have also been evaluated in a recent article.10
Hashman11 suggested that one advantage of subspecialty training in forensic psychiatry is the production of well-prepared clinicians with specific expertise in psychiatry and the law. Moreover, he suggested that these clinicians might be more effective communicators of forensic knowledge within the systems with which they work and among the general public and other professionals with whom they interact. To our knowledge, few published studies have been conducted to examine whether the presence of a fellowship program has had a measurable effect on such outcomes or on other program, clinical, or educational outcomes. Of the few studies that have specifically examined associations between fellowship training and clinical outcomes, some have observed decreased mortality rates in centers that had a critical care medicine fellowship training program,12,13 while others have observed more neutral associations between the presence of trained fellows and clinical outcomes.14 Individual benefits of fellowship training in general have been described by Stern.15 These include better preparation to assume a career in academic medicine, enhanced career satisfaction, and practical and personal rewards related to autonomy in career decision-making.
It is important for educators to be aware of the effects that a fellowship training program has on clinical and educational environments and outcomes. In fact, the ACGME requires new subspecialty programs to declare the way in which they will interact with general programs. While the ACGME is most likely concerned with both positive and negative effects and outcomes, specific questions that must be considered by training directors include whether there is enough clinical activity and faculty to support both programs. The need for such resources is especially true of procedure-dependent specialties.
In addition, the ACGME is interested in educational outcomes via their outcomes project. In 2001, the ACGME Outcome Project began changing the focus of accreditation from educational processes to outcomes. Phase 3 (2006–2011) of the implementation timeline requires training programs to: “Use resident performance data as the basis for improvement and provide evidence for accreditation review.”16 Moreover, the ACGME would like to see external measures of program outcomes as they relate to educational activities, and they further specify the need for programs to use both internal and external indicators for program evaluation and improvement.
In this article, we describe a retrospective examination of general psychiatry residents’ Psychiatry Resident In-Training Examination (PRITE) scores comparing scores during the two-year aggregate period before implementation of a new forensic fellowship training program with those during the two-year aggregate period after the program was implemented, at the same institution. The PRITE is a nationally standardized examination used to assess psychiatric knowledge of resident physicians in training. It is administered each year in most accredited psychiatry residency training programs in the United States. The PRITE is an external criterion that can be used by residency program directors to assess the medical knowledge of individual residents and to provide an indicator of effectiveness of the training program curriculum.17,18
Interface Between the Forensic Training Program and General Adult Training Program
The forensic psychiatry fellowship program at our institution received ACGME accreditation in 2003 and began training fellows in 2005. The primary goal of the program is to educate forensic fellows in all aspects of forensic psychiatry and to prepare them for practice, teaching, research, and system consultation. The program can accommodate up to two fellows per year.
Before the implementation of the fellowship, there was a focused and limited amount of forensic material in the curriculum for general psychiatry residents. For example, the first-year residents received lectures on violence risk assessment and testifying in court, second-year residents on the rights to have or refuse treatment, and third-year residents on malpractice, civil competency, and confidentiality. At that time, the lectures were given by two faculty members in our institution who were board-certified in forensic psychiatry. Limited elective experience was available to residents on forensic hospital units or in other forensic settings before the fellowship, and the residents had limited exposure to the two forensic faculty members. In addition, few residents demonstrated interest in or chose to participate in additional forensic education.
Since the addition of the forensic psychiatry fellowship program at our institution, there have been increased opportunities for meaningful collaboration with the general residency program. For example, our first-year general psychiatry residents (PGY1s) receive lectures on a broader array of forensic topics that include violence risk assessment, confidentiality, forensic case studies, and an introduction to forensic psychiatry; and our third-year general psychiatry residents (PGY3s) receive lectures on topics that include basic law, malpractice, assessment of malingering, forensic case conference, confidentiality, civil competence, and the right to receive or refuse treatment. The lectures are given by the forensic fellows and several faculty members who are either board-certified or board-eligible in forensic psychiatry. Before the implementation of the fellowship, the PGY1 through PGY4 general residents received four to seven hours of didactic instruction on forensic topics during the four years of general training. Since the implementation of the fellowship, the number of hours of forensic-related didactics has increased to 12 for the PGY1 through PGY4 general residents. Forensic fellows and faculty are responsible for presenting didactic instruction to the general residents as part of their yearly lecture series.
In addition to the core forensic lectures that the general psychiatry residents receive, the fellows help conduct Professor Rounds, a didactic program specifically for medical students and PGY1 and PGY2 residents on the inpatient rotation at the state hospital. The program focuses primarily on competence to stand trial, state statutes regarding competence and criminal responsibility, and discussion of forensic cases. In addition, both the faculty and fellows are involved in informal consultation on forensic subjects related to both in- and outpatient general psychiatric practice, including commitment hearings in which junior residents are involved. General residents are encouraged to rotate through forensic services in a fourth-year elective, during which they have opportunities to interact with forensic inpatients, evaluate (under supervision) criminal defendants for both court competency and criminal responsibility, and attend the same forensic didactic lectures available to the forensic fellows. Since the addition of the fellowship, more formalized electives in forensics have become available to PGY4 residents as a result of structured forensic fellowship education programming. Finally, forensic fellows and faculty participate in the departmental grand rounds presentations.
Methods
Data Sources and Procedures
The protocol for this study was classified as exempt and was approved by the Institutional Review Board at our institution. The PRITE has been administered to all general psychiatry residents at our institution for more than 20 years, and the program director archives scores for all residents. In this study, we examined four consecutive years of PRITE scores for all general psychiatry residents. The sampling frame spanned from two years before implementation of the forensic fellowship program (2004–2005) to two years after (2006–2007) it was implemented. This approach yielded 102 PRITE scores from 51 general psychiatry residents over the study period.
Instruments
The PRITE consists of approximately 300 questions that address content in the following areas: growth and development, adult psychopathology, emergency psychiatry, behavioral science and social psychiatry, psychosocial therapies, somatic treatment methods, patient evaluation and treatment selection, consultation-liaison psychiatry, child psychiatry, alcoholism and substance abuse, miscellaneous, geriatric psychiatry, and forensic psychiatry. Scores are reported to the program as two global scores (psychiatry/neurology) and 13 individual subscores for each of the content areas. Although it may change from year to year, we estimate that approximately 15, or five percent, of the total questions in the PRITE pertain to forensic topics. The PRITE has demonstrated reliability and validity in previous research, including content validity,19,20 concurrent validity,21 and predictive validity.22
Analysis
Descriptive statistics, including least-square means and standard deviations, were computed for the PRITE standardized global scores and subscores aggregated for the 2004 and 2005 period (pre-forensic fellowship implementation), and similarly for the 2006 and 2007 period (post-forensic fellowship implementation). A mixed repeated-measures ANOVA model calculated by the commercial software program SAS PROC Mixed version 9.2 was fit to the data. Contrasts were estimated to compare the aggregate mean global and subscores for the post-fellowship time frame versus the pre-fellowship period. This approach took into account the repeated-measures nature of the data, and the contrast was expressed as a t statistic. This analysis was conducted for all 15 scales. There was no difference in the distribution of PGY levels represented in the PRITE scores between the pre- and post-implementation groups (χ2 = 0.200, df = 3, p = .98). Within the pre-implementation time frame, the distribution of scores by PGY level was 29.8 percent PGY1, 25.5 percent PGY2, 23.4 percent PGY3, and 21.3 percent PGY4. Within the post-implementation time frame, the distribution of scores by PGY level was 29.1 percent PGY1, 27.3 percent PGY2, 25.5 percent PGY3, and 19.6 percent PGY4.
Results
Table 1 shows the least-square mean PRITE scores for 2004 and 2005 and for 2006 and 2007. Among the two global scales and the 13 subscales measured by the PRITE, three scales declined slightly, but the decreases were not statistically significant. The remaining scales increased, with five of the scales showing statistically significant improvement. The increase in the score on the Child Psychiatry scale could be considered to be of borderline statistical significance (p = .053). The most dramatic improvements, when stated in standard error units, were in the scores on the Psychiatry Global (t = 9.14), Somatic Treatment Methods (t = 2.64), Alcoholism and Substance Abuse (t = 2.98), and Forensic Psychiatry (t = 3.81) subscales.
PRITE Least-Square (LS) Mean Scores and t-Test Results Comparing General Psychiatry Residents Before (2004–2005) and After (2006–2007) Implementation of the Forensic Fellowship Program
Discussion
The results of this study suggest that the implementation of a subspecialty fellowship program in forensic psychiatry was temporally associated with substantially improved scores on the PRITE forensic subscale for residents in the general training program. Relative to changes on the other 13 subscales, the forensic scale showed the most dramatic improvement in scores across the time periods observed. Specifically, we observed significantly higher PRITE forensic scores among our general psychiatry residents in the two years after implementation of a forensic fellowship than those achieved two years before implementation.
Of interest, other than the significant improvement observed in the Global score, two of the three other significant improvements were in content areas in which subspecialty training was being developed in our department (Addictions Psychiatry and Geriatric Psychiatry). An Addictions Psychiatry fellowship program was implemented at our institution in 2006 and a Geriatric fellowship program in 2007. We also observed significant improvement in the Somatic Treatment Methods subscore and marginally significant improvement in the Child Psychiatry subscore. Although we have no fellowship in the Psychosomatic Psychiatry subspecialty at our institution, we have had a Child and Adolescent Psychiatry fellowship for more than 20 years.
In considering the most likely mechanisms of action to explain these findings, we believe that the improved forensic scores were probably the result of a combination of factors including improved experiences with forensic cases and topics, didactic programming, and interaction with interested and available subspecialty fellows and faculty. Another possible contributing factor during the time frame of this study was that some of the general residents in our program participated in study groups during which they reviewed old PRITE examinations and simulated the examination experience. Given this supplemental study group, one might expect to see improved scores across the board, but in our case it appears that improvements were primarily limited to content areas in which subspecialty fellowship training was emphasized at our institution. Although it is difficult to characterize the full range of interaction that residents have with the forensic program, future research could be undertaken to quantify resident participation in forensic electives, the number of forensic lectures attended, or other indicators of forensic exposure.
In an effort to increase outcomes measurement in residency education, the ACGME requires programs to examine external indicators of program effectiveness. In-training examination scores are an example of an external indicator that can be used for objective program evaluation.17 While our study is somewhat limited in the range of educational outcomes examined, to our knowledge it is the first to examine whether the addition of a fellowship program is associated with in-training examination scores for general residents. The design of the study was observational and does not allow us to make any causal connections between improved PRITE scores and the implementation of fellowship training programs. It could be that the addition of forensic faculty resources and time in general made the difference, or that an increase in didactics made the difference, or even that individual residents sought out electives in forensic psychiatry that are available through the fellowship, thus increasing the scores. While implementation of an ACGME-approved fellowship may help build these components, the design of the study did not allow us to evaluate which factors related to the fellowship were causative of the increased PRITE scores and which were merely correlative. For programs that are not able to implement a fellowship training program, the addition of didactic hours focused on forensics and a faculty enthusiastic and knowledgeable about teaching forensics may also be a way to improve PRITE scores.
This study was limited to four years of data from a single institution. Of interest would be whether the effect observed is demonstrable in all subspecialty programs and at other institutions. In future research, a broader range of topics such as the effects of fellowship training on other educational measures, implementation of standardized examinations, setting clinical competencies, and systems benefits could be explored. Also, when fellowship training includes students in fields outside medicine (e.g., law students), it would be interesting to examine reciprocal effects.
In conclusion, a forensic fellowship training program was associated with significant positive increases in the forensic subspecialty scores on the PRITE for general residents at the same institution. This benefit was realized without additional time spent in forensic clinical rotations and without significant decline in other PRITE content areas. Future studies could build on this work to examine further the outcomes resulting from the interaction between a fellowship and a general residency program. For example, various clinical and educational outcomes could be explored, such as how a fellowship might enrich a program, add expertise within a department, improve teaching efficiency, and create learning and clinical opportunities that enhance the overall training experience.
Acknowledgments
The authors owe a special thanks to Ms. Ashley Lavender for technical assistance and data collection and management.
Footnotes
-
Disclosures of financial or other conflicts of interest: None.
- American Academy of Psychiatry and the Law
