Peer Review to Ensure Quality in Forensic Mental Health Publication

Characteristics of High-Quality Peer Reviewers

If journal editors could select reviewers based on characteristics known to predict high-quality reviews, they could improve the reviews' consistency. Unfortunately, research on the characteristics of superior reviewers is limited and inconclusive. A study of reviewers for the Annals of Emergency Medicine²⁵ showed that most variables tested, such as status as principal investigator on a grant, formal training in critical analysis of statistics, and high academic rank, did not predict high-quality reviews. Relative youth and inexperience and employment at a university-operated hospital favored quality reviews. The predictive power of all variables, however, was weak. Another study also found that younger reviewers produce better reviews.²³ The more time given to reviewing a manuscript, the better the product, up to three hours, after which additional time is not contributory.²⁶ Callaham and Tercier concluded, “… almost none of the experiences and training that might logically be thought to make for a high-quality reviewer … actually predict subsequent performance of high-quality reviews” (Ref. 25, p 0036). Neither can the qualities of a superior reviewer be taught, as indicated by the available research.^25,27,–,29 For example, attendance at a training workshop designed to improve the quality of their reviews had no significant effect on the reviewers.²⁷ The crucial elements that make a reviewer superior remain elusive.²⁷

As an exception to these pessimistic findings regarding the value of training, van Rooyen and colleagues³⁰ achieved favorable results with their Review Quality Instrument (RQI), only after providing guidelines and training in its use. It was not used for peer review, but for rating the quality of the reviewer. More to the point, they found that training in epidemiology and statistics was associated with the quality of reviews (B = 0.201; SEB = 0.05; p < .001). Similarly, Black and colleagues²⁶ found that the only characteristic significantly associated with high-quality reviews was prior training in epidemiology or statistics. In this regard, a special challenge for editors of forensic psychiatry and psychology journals is selecting and recruiting a sufficient number of regular reviewers who have the necessary special expertise in both research methodology and the specific subspecialty subject matter. It may be the case that peer reviewer training is more valuable in some fields or topic areas than in others.

Another exception to the above-cited reports was a study of reviewers of meta-analyses that found remarkable consistency among reviewers in assessing the scientific quality of the submitted papers.³¹ In this study, reviewers were selected on the basis of their experience and interest in research methodology. Oxman and colleagues³¹ posited that the excellent inter-rater agreement was the result of the reviewers' training in research methodology and the explicit rating criteria that were based on process, not outcome. The downside of stressing research methodology to the exclusion of other measures of quality is that worthy dimensions such as the importance of the question to be addressed and accuracy and sophistication of the treatment of the subject matter will be overlooked. Such other dimensions can be most important considerations for forensic articles and analyses of legal topics that are not research reports or meta-analyses.

Reviewers with expertise in scientific methodology should help ensure a satisfactory level of reliability and validity of scientific submissions. Also useful for forensic journals in particular are subject matter experts who are familiar with the field of knowledge and its extant literature. Reviewers need not be in perfect agreement to be useful, as they can make salient observations that complement one another and enhance the value of the composite review.

Research on the characteristics of quality reviewers does not necessarily establish the characteristics of quality reviews. In testing for inter-rater agreement among reviewers, Oxman and colleagues³¹ focused on four criteria for assessment of the quality of the scientific process of meta-analyses, but they acknowledged that other scientific and nonscientific characteristics contribute to the overall quality of scientific articles. Van Rooyen and colleagues³⁰ developed and validated an instrument designed for assessing the quality of manuscript peer reviewers. Those who investigate the characteristics of quality reviewers²⁹ have used this or similar instruments. We will comment later on the characteristics of quality reviews, but for assessment of the peer reviewers or the reviews themselves, such instruments are not entirely satisfactory, having lent themselves to highly subjective valuations. Consider the following item from Callaham's scoring system²⁹ for example: “The reviewer provided the author with useful suggestions for improvement of the manuscript” (Ref. 29, p 2781). No doubt, this is an important feature of many peer reviews, but one without an objective, operationalized standard. Moreover, suggestions for improvement may be disingenuous and unedifying if the submission is actually beyond either redemption or improvement.

Although not tested empirically, the best screening method for superior reviewers would reasonably be through a system of selection and retention that includes an ever-changing pool of ad hoc reviewers in addition to the members of the journal's review board. Regular reviewers are selected on the basis of the consistently high quality of their ad hoc reviews. Editors must monitor the quality of reviews from individual reviewers. Instruments used in research on the quality of reviews may be useful in this regard (e.g., the Review Quality Instrument³⁰), although the routine use of such instruments can be editorially time consuming.²⁵

Selection of Peer Reviewers

In our experience, we suspect that medical journals do not select their reviewers by standardized methods, such as evaluating their expertise in critical review of research or research methodology. Indeed, selection of peer reviewers should be based not only on the quality of their reviews, but on other characteristics, as well, including their established expertise in the desired subject area.

The peer reviewer selection process must first be informed by the composite needs of the journal; if the journal's main focus is forensic psychiatry, a substantial complement of forensic psychiatrists would be appropriate but also with one or more mental health attorneys, forensic psychologists, epidemiologists, or statisticians with advanced expertise in research design. Reviewers might include both generalists within the subspecialty and subject matter experts in the particular areas likely to be emphasized in the journal. The diversity of types of expertise to be represented by the reviewers is determined by the editor's vision for the journal and the interests and needs of the readership to be served.

Beyond regular reviewers, a journal editor may select guest situational or consultant reviewers, who may be depended on to handle a surge of submissions when regular reviewers are already overextended. Ideally, such reviewers are called on for their known specialized expertise in a particular topic or methodology. The use of nonregular reviewers also provides an opportunity for the editor to assess the quality of reviews of potential regular reviewers. Selection of guest reviewers should be based on similar considerations as for regular reviewers except where such reviewers are solicited for expertise not available among the pool of regulars.

A potential reviewer's discipline and areas of interests should be apparent from the curriculum vitae and prior publications, but as emphasized earlier, studies have shown that training, academic rank accomplishments, and prolific publishing record are poor predictors of quality of peer reviews.²⁵ On the other hand, postgraduate training in epidemiology or statistics has been shown to be associated with reviews of good quality.^26,30 Eminence in one's field also helps identify the individual's field. The Thorndike halo³² effect or Matthew effect,^33,34 of inflated impressions of an author's work as judged by a reviewer, may apply as well to an editor's presumption about a well-known author or investigator's ability to provide quality reviews. Editors may select notable experts in a particular field to enhance the prestige of the journal and its marketing success. The finding that academic status and professional reputation do not predict superior peer reviewers may not be reason enough to avoid such appointments. No factor is a strong predictor, and some individuals who are renowned in their fields are also excellent reviewers. Also, it may be more helpful for some submissions to have one review by someone with a high level of expertise in the subject matter and another by a reviewer who is otherwise of superior quality but without unique knowledge of the subject matter. Our knowledge of what determines a quality peer review is limited.

Some journal editors offer the authors the opportunity to nominate or suggest potential reviewers for their manuscripts. An author's recommendations can help to identify experts in a subject for which the editor's known choices are limited. They can also assist the editor in expanding the pool of reviewers. On the negative side, they can give the author an opportunity to select a reviewer who will be lenient or one with an undisclosed conflict of interest. Reducing the level of blinding and objectivity in selecting reviewers may diminish actual or perceived fairness. To our knowledge, all of these questions, as well as the overall success and utility of this approach, have not been evaluated empirically.

Beyond having expertise in the field and proficiency in conducting reviews of high quality, the desirable reviewer should be available and responsive. He should be willing and able to complete reviews in a timely manner. The superior reviewer also strives for fairness and impartiality and submits reviews that are constructive and free of degrading, abrasive, sarcastic, competitive, and gratuitously critical remarks.¹ The recommendation as to whether the manuscript should be published should not be shared with the author. Unless selected for a specific purpose such as assessing the appropriateness of the statistical method used in the research, it is essential that reviewers be familiar with the professional area that is the subject of the manuscript as well as the relevant scientific literature on that subject and the weaknesses of the extant literature.

Recruitment and Retention of Peer Reviewers

Regardless of how excellent in quality an individual's reviews can be, if the reviewer does not perform the review, it is of no use. The timeliness of submission is also important. Although in some cases a late review is better than none, delay can result in a shortened time frame for the editor and for the authors if revision is needed and can cause the submission to fall off the publication track. Thus, useful reviewers complete their work in a timely manner.

Many potential reviewers either do not agree to review a given manuscript or simply do not respond to the soliciting editor. Research by Pitkin and Burmeister^35,36 showed that requesting a potential reviewer in advance leads to an increased risk that the request will be rejected. Sending the manuscript to the reviewer without first seeking permission results in a higher rate of review. Those reviewers who agreed in advance, however, complete their reviews more promptly.³⁵ When reviewers were late, reminders were effective in prompting completion within seven days in about two-thirds of late reviews. This improved responsiveness with a reminder occurred regardless of whether the reminder was delivered by telephone, fax, or e-mail.³⁶

Peer reviewers report many reasons for agreeing to review a scientific manuscript. According to Tite and Schroter, the most common reasons were that the paper seemed to make a contribution to the subject area, the topic pertained to the reviewer's own interest, and the review provided an opportunity for the reviewer to learn something new from the manuscript (Ref. 37, p 10).

Although the possibility has not been examined empirically to our knowledge, some individuals may be willing reviewers if named to the journal's editorial board, but remain unwilling without such acknowledgment. Other expressed reasons for both regular and guest reviewers to decline are time constraints and insufficient expertise on the subject of the manuscript.

It is impractical to appoint all who would review on an ad hoc basis to the editorial board. Similarly, editors may wish to call on a reviewer who has rotated off the board, but once off the board or once in professional retirement, some would-be reviewers no longer desire to review. Publication and journal editors commonly reward guest reviewers by listing their names with appreciation in the final issue of the volume, but the incentive value of this measure is unknown.

Desired Qualities of Peer-Reviewed Ratings

If peer review is to serve well its two-fold raison d'être, to select submissions worthy of publication and to improve the quality of papers before publication, attention must be directed to those qualities of peer review assessments that best support these purposes. Naturally, scientific reports should be examined for their scientific quality, to ensure reliability and validity and to diminish the risk of bias. Ensuring quality means attending to methodology and components of the process of designing and executing the study, including the concept of the research, subject recruitment, consistency testing, validity testing, appropriateness of methods, representativeness of the condition that is studied,³¹ and conflicts of interest. A full exposition on the analysis and the quality of a scientific study is beyond the scope of this review, except to note that certain types of studies require additional methodological and ethics-related considerations. For example, review of a randomized drug trial report might best be accomplished using the 30-item scale based on the Consolidated Standards of Reporting Trials (CONSORT)³⁸ or a reasoned modification of this scale.³⁹

The scientific process or methodology of the research or meta-analysis is not the only consideration in assessing the scientific quality of a manuscript. Features other than scientific methodology and data-analysis strategies¹⁶ include the manuscript's potential impact on future scientific developments, the degree of innovation in the approach, the author's imagination and originality,¹⁶ the importance of the question that is addressed,²⁸ and the relevance and completeness of the literature review.¹⁶ Additional considerations encompass the manuscript's clinical relevance and literary quality³¹ (including clarity of written expression,⁴⁰ organization of information,¹⁶ and style that is appropriately measured, focused, and based on evidence presented or cited); acknowledgment of limitations; and well-reasoned presentation.

Of the several instruments developed to assess the quality of reviews, only that designed by van Rooyen and colleagues,³⁰ who themselves recognize that there is no gold standard for criteria validation and comparison, has been empirically tested for internal consistency and reliability. The Review Quality Instrument (RQI) showed high internal consistency with a Cronbach's α of .84 and good test–retest (κ_w = 1.00) and inter-rater (κ_w = .83) reliability (weighted κ statistic). Parameters incorporated into this instrument and therefore deemed useful in assessing the quality of a manuscript were importance of the research question, originality, methodological strength with minimal weaknesses, and presentation of the results. Characteristics of a quality review itself and not the manuscript are constructiveness and substantiation of written comments intended for the author(s). Although it is not always mentioned in studies of peer review, we believe that substantiation of the comments in the review is just as important as any criticism.

In the study by van Rooyen and colleagues,³⁰ small improvements in the level of internal consistency resulted from removing the items on originality and importance. However, these items were retained in the instrument because of their importance to editors. Indeed, we would add that an item may be no less relevant to quality simply because its internal consistency or reliability has not been empirically demonstrated. In some instances, the item can be reworded to achieve greater consistency and reliability. In any case, the valuation of specific parameters should be most useful to the editor and to the authors as suggestions for improvement, if examples are cited to substantiate the specific rating or comment.

The identification of the desired qualities of peer review assessments presupposes identification of quality parameters of the manuscripts themselves. Dougherty and colleagues⁴¹ conducted a 69-item online survey of 1,675 nursing journal peer reviewers, to identify which indicators of quality were most favored by reviewers. The most critical factors in assessing the value of a manuscript were whether the topic of the manuscript is of current interest, whether it represents a newly emerging area, and whether the manuscript contributes to knowledge or research evidence. Clinical relevance and research rigor also contributed to a manuscript's publication value. Thus, beyond its scientific vigor, the valued manuscript is also of interest to the journal's readership and is of material value to the knowledge base of the field.

In emphasizing the importance of inter-reviewer reliability, we want to add a word about creative disagreement. One reviewer may detect a flaw or suggest an improvement that does not occur to other reviewers,¹⁶ resulting in inter-reviewer unreliability that is not undesirable. Though again untested, editors may constructively seek some divergence in types of expertise for a given manuscript without blindly pursuing maximum reliability. Such diversity of critical input can be of special, if unmeasured, importance for cross-disciplinary manuscripts, as many are in the interdisciplinary field of psychiatry and the law. Reviewers can and do have different preferences as to manuscript content and approach, and these differences can influence their ratings of manuscript quality and acceptability.

Another approach to assessing the value of peer review in improving the quality of manuscripts is to assess how they are changed from initial submission to final publication. When such a study was conducted concerning manuscripts for the Annals of Internal Medicine, the changes responded to the following types of initially identified problems: too much information (too detailed, redundant, extraneous), too little or missing information (not enough detail, emphasis needed), inaccurate information (incorrect, inappropriately synthesized, inadequately synthesized), misplaced information (alternative location preferred, inappropriate location), and structural problems.⁴² Of these, changes were most frequently made because information was missing or extraneous.

As with other studies of peer review, these findings may or may not be extrapolated to forensic mental health journals. The extent to which these changes were driven by journal policy or judgment of the editor, peer reviewers, or both is unclear. Important to consider, however, is that the peer review process can and perhaps often does result in substantial changes, assumed to be improvements, in the quality and quantity of information. Such changes need not conform in all respects to the scientific quality of the manuscript per se, and the inter-referee reliability for the need for such changes in information is unknown.

Is Peer Review a Worthwhile Enterprise?

Is peer review a worthy enough endeavor to continue? Certainly, the two-fold aims of peer review are worthy ones: winnowing the wheat from the chaff in terms of publishable science and other scholarly work and improving manuscripts before publication to ensure satisfactory quality. There seems to be a consensus in the literature that peer review is the desired method for selection and quality improvement and that the scientific community currently has no viable alternative to its use. The literature that assesses peer review on the other hand also suggests that it does not uniformly meet these aims as well as it should, is costly in professional time and resources, and can in some ways be counterproductive; but, in fact, those journals that are not peer reviewed are often regarded less favorably.

In addition to the two-fold explicit purposes of peer review, it ought to support the perception of and bona fide presence of fairness by striving for it. Just as courts strive for fairness and expert witnesses strive for objectivity, peer review should strive for balanced assessments. With a peer review process in place, journal readers should have confidence that editorial selections are properly informed and based on merit. Authors and investigators who submit manuscripts for publication should also be reassured by the existence of a peer review process that selection is determined more by the merit of the submission than the uninformed whims and biases of a single editor. Similar to sound selection and quality improvement, however, fairness is enhanced by a process that works well and can be undermined by flawed procedures. The appearance and the actual presence of fairness are both requisite elements.

Thus, the aims of peer review—quality selection, quality improvement, and enhanced fairness—are desirable. There seems to be no better approach that can replace peer review, regardless of its imperfections. What must be addressed then is the matter of improving the peer review process, so as to ensure that it usefully achieves its aims.

Improving the Quality of Peer Review

Without attention to improvement, any human process risks unintended regression with worsening results. Improvement measures must be practical, however, and achievable without counterproductive expenditure of time and energy. Some authors have listed methods, often speculative, for improving peer review. We will limit our attention to potential improvement measures that address real or potential flaws identified in this review with special consideration for the unique needs and circumstances of journals in forensic psychiatry and psychology.

Studied attention should be given to the positive and negative value of reviewers' comments for the author, for whom they are arguably more influential, at least in the process of manuscript improvement, than the parameter ratings. Yet the individual comments are less studied and less subject to quantitative comparisons. It may be that the variety of comments without inter-rater reliability are more useful than carbon copy reviews.

Before venturing to suggest measures for improving peer review of forensic psychiatry and psychology journals, some caveats and limitations must be emphasized. If weaknesses, inconsistencies, and shortcomings in achievement of desired results cause some to doubt the value of peer review, published recommendations for improvement are even less empirically studied and substantiated. Idealistic measures for improvement can be impractical; the more onerous the measures for editors and reviewers, who typically are fully occupied with responsibilities of employment (clinical, forensic, administration, and research), the less likely the measure or the review itself will be accomplished. There may be financial or other practical limitations in available resources that preclude institution of peer reviewer quality improvement measures. Commercial journal publishers may not desire to institute changes in just one of their published periodicals. Improvement measures should be especially relevant to forensic psychiatry and psychology journals that have not been included in the literature on peer review:

• The journal should select reviewers based on their expertise and experience in writing and editing in forensic mental health. Examining the reviewer's curriculum vitae and examples of published or edited works can be useful in this regard. Familiarity with the special journal is essential, so that the reviewer can thereby gauge the journal's interests and standards. Of importance, assessing a reviewer's performance as a guest reviewer is a reasonable attempt at identifying his potential work product as a reviewer. Some variety in the skills and knowledge of the entire group of regular reviewers is desirable (e.g., health law attorneys and law professors, forensic psychologists, forensic psychiatrists, forensic social workers, experts in research methodology, and statistical analysis) is more useful than having everyone come from one of these fields. New reviewers should describe their areas of expertise and the limits thereof to the editor.

• The journal should provide regular reviewers with written guidelines that explain how the manuscript rating form is to be used,²⁷ pitfalls to be avoided, and important qualities of good reviews. Editors can provide examples of improper reviewer conduct to potential manuscript reviewers, such as publicly discussing or citing a reviewed manuscript before its acceptance and publication or using the manuscript for the reviewer's own work. These written guidelines can advise reviewers in advance that they will receive periodic feedback on the quality of their reviews and suggestions for improvement. The journal should have those selected to be reviewers confirm in writing their understanding of their obligations.

• The journal should design the manuscript review form to include clearly written, practical measures of manuscript quality, using those already recognized in the peer review literature as well as items specific to the subspecialty of forensic mental health publications. The journal should require commentary that substantiates the ratings, particularly the critical ratings, with corresponding suggestions for improvement of the manuscript.

• Guest reviewers should be rewarded in any nonfinancial way that is feasible. At a minimum, they should be sent a letter of appreciation, recognized in an issue of the journal, and provided with a copy of the journal that contains the article that they reviewed.

• The journal should adopt a mechanism for monitoring the performance of regular reviewers and providing them with periodic feedback. These comments should include specific recommendations for improvement and commendation on superior aspects of their reviews and expression of appreciation for their conscientious, diligent work.

• Editors can provide peer reviewers with copies of reviews of the same manuscript by other reviewers. Reading the work of others may enhance the reviewers' motivation to perform comprehensive reviews and may serve a training function, especially for reviewers who tend to be outliers in their reviews.

• The journal should periodically review appointments of regular reviewers, perhaps every four years, but stagger the turnover. Reappointment should be based on satisfactory peer review performance determined through a methodical and perhaps empirical performance review procedure. Options include replacing all regular reviewers within a specified period of tenure, retaining those who reliably provide timely reviews of high quality and replacing those whose work is less exemplary or vital, or designating as consultants some of the previously regular reviewers.

Because of the difficulty in extrapolating findings that may be unique to other disciplines to forensic psychiatry and forensic psychology, research on the peer review process in forensic journals would place any recommendations for change or improvement, such as these, on an empirically firmer and more useful footing.