Abstract
In the 2019-2020 academic year, there were 48 accredited forensic psychiatry fellowship programs in the United States. Programs vary in application requirements and timeline. There are no published objective data on factors that fellowship program directors (PDs) use when selecting fellows. We created an electronic survey that was emailed to PDs via a list from the Association of Directors of Forensic Psychiatry Fellowships. The survey was open November 6, 2019 to December 31, 2019. Twenty-five PDs participated from programs ranging in size from one to six positions, receiving zero to 30 applications. The most important factors when selecting a candidate to interview were “perceived commitment to specialty” and “perceived interest in your program.” The most important factors when offering a position were “interpersonal skills” and “interactions during interview.” The least important factors in both categories were USMLE/COMLEX scores and honor society membership(s). “Lack of a set timeline” during the application process was the most frequently cited difficulty (via multiple choice) during the application and interview process. Our study is the first to provide quantitative data regarding factors that forensic psychiatry fellowship PDs use to evaluate applicants in decisions regarding offering interviews and positions.
Forensic psychiatry is defined by the Accreditation Council for Graduate Medical Education (ACGME), the accreditation body for graduate medical education in the United States, as the psychiatric subspecialty that “focuses on interrelationships between psychiatry and the law …”.1 From academic year (AY) 2014–2015 to AY 2019–2020, the number of ACGME-accredited forensic psychiatry fellowship programs in the United States has grown 20 percent, from 40 to 48.2 In the 2018–2019 AY there were 66 active fellows in those programs3 with an estimated 68.8 percent of positions in forensic psychiatry fellowship programs going unfilled.4 This comes during a time when the number of general psychiatry residency programs have grown significantly, and those programs have become more sought-after and competitive.4,5 At the same time, as the increase in competitiveness of general psychiatry residency programs, psychiatry subspecialty programs in the United States have seen a decline in applicants and an increase in unfilled positions.6 Several factors may explain the difficulty in recruiting subspecialty applicants, including lack of a standardized application process, variability in resident experiences in forensic psychiatry,4 and desire to start practicing.
The purpose of the current study is to investigate objective and subjective factors forensic psychiatry fellowship program directors (PDs) use when evaluating applications and selecting fellows for their program. Our study also evaluates (albeit in a less rigorous manner) the difficulties PDs encounter when selecting fellows for their respective programs. Prior surveys and studies conducted by the National Resident Matching Program (NRMP)7 and the American Association of Medical Colleges (AAMC)8 have included PDs from primary specialties, including general psychiatry. Forensic psychiatry fellowship programs, however, have not been examined. Sub-specialty surveys have also been conducted, including one of pediatric otolaryngology fellowship PDs.9 The present survey is the first to investigate and present data from forensic psychiatry fellowship PDs.
Methods
In October 2019, a survey was created using the SurveyMonkey platform. The survey was informed by a review of the literature, evidence from past similar surveys,7,–,9 and factors relevant to a fellowship application. The application factors were selected based on our review of previous PD surveys, notably those by the NRMP7 and AAMC.8 In addition, we added factors (e.g., writing samples) that are included in forensic psychiatry fellowship applications10 but are not included in previous surveys. The entire survey is available in the Appendix. The study was reviewed and approved by the UConn Health Institutional Review Board as nonhuman subject research. The survey was then presented to forensic psychiatry fellowship PDs by the first author (EK) and discussed in-person at the annual meeting of the Association of Directors of Forensic Psychiatry Fellowships (ADFPF) on October 23, 2019. A link to the survey was emailed to PDs on November 6, 2019 using an email list provided by the ADFPF. Reminder emails were sent to PDs on three subsequent dates to improve response rate. The survey could be completed on mobile (e.g., smartphone, tablet) or desktop devices. It could be completed in one sitting or could be saved so the user could complete it at a later time. The survey was closed on December 31, 2019. Data were exported to the UConn Health server for storage and analysis. PDs were informed that responses (either subjective or objective) would be reported only in aggregate form to prevent individual identification of program responses. Any question on the survey could be skipped and respondents were allowed to proceed to subsequent sections. This was done to allow PDs to skip any questions that they did not want to answer for various reasons, such as not wanting to reveal their program or specific aspects of their program.
Data Analysis
Because of the descriptive nature of this first survey about criteria that forensic psychiatry fellowship PDs used to select applicants, descriptive statistics of survey items were reported: mean and standard deviation for count and rating items and n (%) for dichotomous check list items.
Results
Program Characteristics
Of the 48 accredited forensic psychiatry programs in the United States at the time of this survey study, 25 (52%) participated. Programs varied in number of fellowship positions available, from 1 to 6 positions (M [SD] = 2.8 [1.3]). Programs received zero to 30 applications (M [SD] = 9.3 [6.9]) with 3.3 (2.2) applicants per position. PDs invited between zero and 13 applicants for an interview (M [SD] = 5.4 [3.6]) with 2.0 (1.3) invitations per position and interviewed between zero and 12 applicants (M [SD] = 4.6 [3.3]) with 1.6 (1.2) interviews per position. PDs offered a range of zero to six applicants a position in their program (M [SD] = 3.0 [1.8]) with 1.0 (.4) offers per position, and zero to six applicants accepted the offer that was given to them (M [SD] = 2.4 (1.7)] with .8 (.4) acceptances per position. The distribution of these raw count items is displayed in Figure 1 to illustrate the diversity of forensic psychiatry programs.
Program characteristics histogram.
The survey took approximately 15 min to complete and over 90 percent of respondents completed it in under 20 min. One PD skipped all 29 interview-related items, four PDs skipped all 32 position-related items, including the one who skipped all 29 interview-related items.
Difficulties in Selecting Fellows
In the survey, PDs were asked, “Which of the following [ten] statements describe(s) difficulties your program experiences in selecting fellows?” Respondents could choose up to three statements and were offered an “Other” choice that allowed PDs to enter text into a textbox.
Difficulties Encountered by Program Directors When Selecting Fellows
Table 1 displays the ordered list of difficulties encountered by PDs during the selection process, from the most endorsed to the least endorsed. A significant number of PDs (52%) cited “lack of a set timeline that other programs have for applications, interviews, offers” as a difficulty when selecting fellows. Although the “Other (please specify)” option was the second most common response, program director comments were generally variations of other response options. For example, two PDs wrote “Many fellowships chasing few fellows” and “Too many fellowship spots, not enough fellows” which would fit with the “Small number of applications” option. Other responses followed a similar pattern.
Factors When Deciding Whom to Interview
Twenty-nine factors were rated on a scale of 0 to 10, with 0 being not important at all (e.g., not considered) and 10 being the most important. Figure 2 lists nine factors that were considered important by PDs when deciding whom to interview, with a mean rating of 7 or higher. “Perceived commitment to specialty” had the highest M rating of 8.4 (SD = 1.3). Nine (38% out of 24) PDs rated “Evidence of professionalism and ethics” as the most important factor (score 10) and it shared the second highest mean rating with “Perceived interest in your program”: M (SD) = 8.2 (2.3), and 8.2 (1.4), respectively.
Factors rated as important when deciding whom to interview (mean rating of 7.0 or higher).
Figure 3 lists eight factors that were considered to be not important by PDs when deciding whom to interview, with a mean rating of 4 or lower. Factors considered important for general psychiatry selections7 (e.g., USMLE Step 1 and 2 scores, graduation from U.S. medical schools) were not considered as important by forensic psychiatry PDs when deciding whom to interview. Notably, factors such as USMLE Step 3 scores and PRITE scores were also not considered as important by forensic psychiatry PDs. Factors that received the lowest scores (0 or 1) the greatest number of times included USMLE Step 1 scores (35%, eight of 23 PDs), Step 2 CK scores (35%, eight of 23), Step 2 CS scores (35%, eight of 23), and Step 3 scores (26%, six of 23).
Factors rated as not important when deciding whom to interview (mean rating of 4.0 or lower).
Factors Used in Deciding to Offer a Position
Thirty-two factors were rated on a scale of 0 to 10, with 0 being not important at all (e.g., not considered) and 10 being most important. Nine PDs considered “Perceived interest in your program” (43%, nine of 21), “Evidence of professionalism and ethics” (43%, nine of 21), and “Interactions with faculty during interview and visit” (45%, nine of 20) to be the most important factors.
Figure 4 lists ten factors that had a mean rating of 7 or higher. “Interpersonal skills” had the highest mean rating (9.1, SD = 1.1), followed by “Interactions with faculty during interview and visit” (8.9, SD = 2.1). “Perceived interest in your program” and “Evidence of professionalism and ethics” also received high mean rating (8.7, SD = 1.3 and 8.7, SD = 1.7, respectively).
Factors rated as important when deciding to offer a position (mean rating of 7.0 or higher).
Figure 5 lists eight factors with mean rating of 4 or lower. Notably, PRITE and USMLE/COMLEX scores, including Step 3 scores, were rated low. Graduate of non-U.S. medical schools was also not an important factor in offering the fellowship position.
Factors rated as not important when deciding to offer a position (mean rating of 4.0 or lower).
Objective factors such as test scores (PRITE, USMLE Step 1/COMLEX Level 1, USMLE Step 2 CS/COMLEX Level 2 PE, USMLE Step 2 CK/COMLEX Level 2 CE) were rated 0 by five forensic psychiatry PDs. When a rating of 0 and 1 are combined to form one category, USMLE Step 1/COMLEX Level 1 score, USMLE Step 2 CK/COMLEX Level 2 CE score, and USMLE Step 2 CS/COMLEX Level 2 PE score were all selected nine times. USMLE Step 3/COMLEX Level 3 score was selected eight times.
Discussion
This survey is the first to systematically measure the qualities that forensic psychiatry fellowship PDs use to evaluate applications to their programs, both when inviting an applicant to interview and offering a position in their program. In the 2018 NRMP Program Director Survey,7 the overall response rate was 29.3 percent and the response rate for general psychiatry was 29.8 percent. The 2016 AAMC Program Director Survey8 had a combined response rate of 39 percent and a response rate from psychiatry of 40 percent. This survey of forensic psychiatry fellowship PDs garnered a 52.1 percent response rate (25 of 48).
Other surveys of PDs7,8 studied primary residency programs (e.g., anesthesiology, internal medicine, neurology, ob/gyn, surgery, pediatrics), while few have studied subspecialties.9 In the NRMP Program Director Survey,7 91 percent of psychiatry residency PDs cited USMLE Step 1/COMLEX Level 1 score as a factor in selecting applicants to interview, with an average rating of 3.6 of 5 (ratings on a scale from 1 = not at all important to 5 =very important). The Medical Student Performance Evaluation (MSPE/Dean's Letter) was also cited by 91 percent of PDs with an average rating of 4.3. The factor with the highest average rating from psychiatry residency PDs in selecting applicants to interview was “applicant was flagged with Match violation by the NRMP” (average rating = 4.9) followed by “evidence of professionalism and ethics” (average rating = 4.5). When ranking applicants in the Match, “interactions with faculty during interview and visit,” “interactions with housestaff during interview and visit,” and “interpersonal skills” all received an average rating of 4.9 of 5.
Overall, forensic psychiatry PDs appear to value more subjective qualities, such as perceived commitment to forensic psychiatry, perceived interest in the program, and evidence of professionalism and ethics. Some categories included in our survey, such as PRITE scores, are not applicable to general psychiatry residencies, while other categories, such as Dean's Letter/MSPE or a Match violation, are not directly applicable to fellowships. An application component unique to forensic psychiatry compared with general psychiatry is the writing sample (e.g., admission note, forensic evaluation, published article), which 73 percent of programs require.10 This is distinct from a personal statement, which 94.6 percent of programs require.10 Forensic psychiatry as a practice requires a significant amount of writing, which likely explains why so many fellowship programs require one (and often two) writing samples in an application.
Some of the more subjective qualities (e.g., perceived commitment to specialty) can be displayed in other parts of the application (e.g., personal statement) and can be demonstrated through electives in forensic psychiatry during residency, attending the AAPL conference, and presenting or publishing on a topic related to forensic psychiatry. Some qualities (e.g., perceived interest in a particular program) can be assessed during an interview. While USMLE scores or medical school grades (both objective in nature) are straightforward to convey in an application, other qualities may require more nuanced evaluation. This knowledge is important for applicants and can enhance their overall application. A more comprehensive survey in the future can identify how PDs specifically assess these factors.
This survey did not explicitly differentiate between factors that PDs use to evaluate internal versus external candidates. We recommend that it be a consideration for future study, however, for two reasons. First, the factor of being an “internal” candidate (i.e., applying to a fellowship program at the same institution where the applicant is completing training prior to forensic psychiatry) had a mean rating of 7.3 (SD = 2.3) when PDs consider a position offer. At a minimum, this suggests that PDs consider status as an internal candidate (Fig. 4). Second, the interview timeline agreed upon by the ADFPF exempts internal candidates from agreed-upon timelines, which allows programs to offer positions to applicants within their own general programs at any time.11 These factors suggest that an internal candidate applicant may be viewed as unique, which applicants should note. Therefore, the authors recommend further inquiry.
The higher value placed on subjective criteria by forensic PDs contrasts to the relative importance of objective criteria when applying to a general psychiatry residency. We surmise that once an applicant is at the stage of applying to a subspecialty fellowship, test scores and grades are satisfactory and relatively equivalent. Instead, it appears that PDs value applicants who are truly interested in forensic psychiatry in general, in their program in particular, and who demonstrate strong interpersonal skills.
As noted in the Results, the most frequently cited difficulty encountered by PDs when selecting fellows was “lack of a set timeline that other programs have for applications, interviews, and offers.” At the time of the survey, it was common practice for some programs to offer positions before other programs began interviewing candidates or offering positions. This was done by PDs to ensure that positions in their program were filled. As a result, applicants were unable to evaluate all of their options before deciding which program to ultimately attend. These pressures created a situation in which the application process would shift earlier and earlier each year. Therefore, an area that the authors believe would benefit from increased transparency and guidance is the application timeline. Starting January 1, 2020, the ADFPF implemented guidelines11 that standardize when programs can begin conducting interviews and when they can begin extending position offers. This is an important step toward alleviating the concerns noted by many PDs in this survey. In addition, the ADFPF released a common application that programs can implement in their selection process. The common application is an additional measure that can improve the application process for programs and applicants alike.
Limitations
This is a preliminary study to generate data regarding the selection process of forensic psychiatry fellows. While a response rate of 52 percent of forensic psychiatry fellowship PDs is higher than that of the NRMP survey of PDs (29.3% response rate), the data would still be stronger if more PDs provided their input. Another limitation is that our survey was conducted before forensic psychiatry PDs implemented an application timeline and a common application. The results of a subsequent survey with similar questions would likely yield different results in light of these developments, with other concerns becoming more relevant.
Conclusions
This study is the first to investigate objective and subjective factors that forensic psychiatry fellowship PDs use when evaluating applications to their programs and selecting fellows. Our study demonstrates a clear difference in factors considered important when compared with the general psychiatry residency application process. Objective factors such as standardized test scores and honor society membership were among the lowest rated factors by forensic psychiatry PDs, while being considered salient features of the general psychiatry residency application.
A recent review by Richard Frierson, MD4 highlights the historical lack of structure and organization in the forensic psychiatry fellowship application process: “Each program has its own application requirements, its own timeline for interviewing applicants, and its own selection process” (Ref. 4, p 22). In addition, forensic psychiatry PDs have historically not come to a consensus to participate in The Match®. Recent work by members of the ADFPF to implement a common application12 for forensic psychiatry fellowships and encourage a timeline for programs to offer positions11 should alleviate some of the concerns expressed historically by PDs and noted in this study.
As Dr. Frierson noted in his American Academy of Psychiatry and the Law (AAPL) Presidential Address on its 50th anniversary, “The future of forensic psychiatry training is bright” (Ref. 4, p 23). The authors of this study agree and believe that we can work toward this goal by continuing to examine various aspects of forensic training, including the application process and selection of fellows. Creating an application timeline11 and providing data-driven insights into the selection process will benefit applicants and fellowship programs, strengthening both and ensuring a brighter future for our field. This survey constitutes another important step toward achieving these goals.
Acknowledgments
We gratefully acknowledge the assistance and support of Richard Martinez, MD, MH, President of the Association of Directors of Forensic Psychiatry Fellowships.
APPENDIX
The Forensic Psychiatry Fellowship Program Director Survey
[Note: Formatting of the survey below varies from the formatting participants saw on SurveyMonkey. All questions remain the same.]
Please answer the following questions with a number that applies to the most recent complete application cycle in your forensic psychiatry fellowship program.
How many fellowship positions are available in your program?
How many applications did you receive?
How many candidates did you invite for an interview?
How many candidates did you interview?
How many candidates did you offer a position to?
How many candidates accepted a position?
Which of the following statements describe(s) difficulties your program experiences in selecting fellows?
(Select up to 3 choices)
It is difficult to compare information across different residency programs
Lack of a set timeline that other programs have for applications, interviews, offers
Large number of applications
Small number of applications
Lack of reliable information about an applicant's personal characteristics
Lack of reliable information about an applicant's academic/technical preparation
Lack of information about which applicants are genuinely interested in my program
Applicants cancel interviews
Application/interview season is too long
Application/interview season is too short
Other (please explain below)
On a scale of 0 (zero) to 10 (ten), with 0 being not important at all (e.g., not considered) and 10 being the most important, please rate each of the following factors when deciding whom to invite to interview at your program.
USMLE Step 1/COMLEX Level 1 score
USMLE Step 2 CK/COMLEX Level 2 CE score
USMLE Step 2 CS/COMLEX Level 2 PE score
USMLE Step 3/COMLEX Level 3 score
Any failed attempt in USMLE/COMLEX
Letters of recommendation
Medical Student Performance Evaluation (MSPE/Dean's Letter)
Medical school grades
Personal statement
Writing sample(s)
PRITE scores
Perceived commitment to specialty
Perceived interest in your program
Personal prior knowledge of the applicant
Evidence of professionalism and ethics
Leadership qualities
Alpha Omega Alpha (AOA) membership
Gold Humanism Honor Society (GHHS) membership
Other awards or special honors
Other life experience(s)
Volunteer/extracurricular experiences
Lack of gaps in education/training (e.g., time off during/after medical school and/or residency)
Graduate of highly-regarded U.S. medical school
Graduate of non-U.S. medical school (e.g., IMG)
Resident in your institution's psychiatry residency program (i.e., an internal applicant)
Graduate of highly-regarded residency program
Demonstrated involvement and/or interest in research
Demonstrated interest in academic career
Visa status
9. On a scale of 0 (zero) to 10 (ten), with 0 being not important at all (e.g., not considered) and 10 being the most important, please rate each of the following factors when deciding whom to offer a position at your program.
USMLE Step 1/COMLEX Level 1 score
USMLE Step 2 CK/COMLEX Level 2 CE score
USMLE Step 2 CS/COMLEX Level 2 PE score
USMLE Step 3/COMLEX Level 3 score
Any failed attempt in USMLE/COMLEX
Letters of recommendation
Medical Student Performance Evaluation (MSPE/Dean's Letter)
Medical school grades
Personal statement
Writing sample(s)
PRITE scores
Perceived commitment to specialty
Perceived interest in your program
Personal prior knowledge of the applicant
Evidence of professionalism and ethics
Leadership qualities
Alpha Omega Alpha (AOA) membership
Gold Humanism Honor Society (GHHS) membership
Other awards or special honors
Other life experience(s)
Volunteer/extracurricular experiences
Lack of gaps in education/training (e.g., time off during/after medical school and/or residency)
Graduate of highly-regarded U.S. medical school
Graduate of non-U.S. medical school (e.g., IMG)
Resident in your institution's psychiatry residency program (i.e., an internal applicant)
Graduate of highly-regarded residency program
Demonstrated involvement and/or interest in research
Demonstrated interest in academic career
Visa status
Interactions with faculty during interview and visit
Interpersonal skills
Other post‐interview contact
10. Please provide any comments you feel are important or relevant and were not covered by this survey.
End of survey
Footnotes
Disclosures of financial or other potential conflicts of interest: None.
- © 2021 American Academy of Psychiatry and the Law
