Context: Selecting peer reviewers who will provide high-quality reviews is a central task of editors of biomedical journals.
Objectives: To determine the characteristics of reviewers for a general medical journal who produce high-quality reviews and to describe the characteristics of a good review, particularly in terms of the time spent reviewing and turnaround time.
Design, setting, and participants: Surveys of reviewers of the 420 manuscripts submitted to BMJ between January and June 1997.
Main outcome measures: Review quality was assessed independently by 2 editors and by the corresponding author using a newly developed 7-item review quality instrument.
Results: Of the 420 manuscripts, 345 (82%) had 2 reviews completed, for a total of 690 reviews. Authors' assessments of review quality were available for 507 reviews. The characteristics of reviewers had little association with the quality of the reviews they produced (explaining only 8% of the variation), regardless of whether editors or authors defined the quality of the review. In a logistic regression analysis, the only significant factor associated with higher-quality ratings by both editors and authors was reviewers trained in epidemiology or statistics. Younger age also was an independent predictor for editors' quality assessments, while reviews performed by reviewers who were members of an editorial board were rated of poorer quality by authors. Review quality increased with time spent on a review, up to 3 hours but not beyond.
Conclusions: The characteristics of reviewers we studied did not identify those who performed high-quality reviews. Reviewers might be advised that spending longer than 3 hours on a review on average did not appear to increase review quality as rated by editors and authors.