Skip to main content
  • Research article
  • Open access
  • Published:

Risk stratification using data from electronic medical records better predicts suicide risks than clinician assessments

Abstract

Background

To date, our ability to accurately identify patients at high risk from suicidal behaviour, and thus to target interventions, has been fairly limited. This study examined a large pool of factors that are potentially associated with suicide risk from the comprehensive electronic medical record (EMR) and to derive a predictive model for 1–6 month risk.

Methods

7,399 patients undergoing suicide risk assessment were followed up for 180 days. The dataset was divided into a derivation and validation cohorts of 4,911 and 2,488 respectively. Clinicians used an 18-point checklist of known risk factors to divide patients into low, medium, or high risk. Their predictive ability was compared with a risk stratification model derived from the EMR data. The model was based on the continuation-ratio ordinal regression method coupled with lasso (which stands for least absolute shrinkage and selection operator).

Results

In the year prior to suicide assessment, 66.8% of patients attended the emergency department (ED) and 41.8% had at least one hospital admission. Administrative and demographic data, along with information on prior self-harm episodes, as well as mental and physical health diagnoses were predictive of high-risk suicidal behaviour. Clinicians using the 18-point checklist were relatively poor in predicting patients at high-risk in 3 months (AUC 0.58, 95% CIs: 0.50 – 0.66). The model derived EMR was superior (AUC 0.79, 95% CIs: 0.72 – 0.84). At specificity of 0.72 (95% CIs: 0.70-0.73) the EMR model had sensitivity of 0.70 (95% CIs: 0.56-0.83).

Conclusion

Predictive models applied to data from the EMR could improve risk stratification of patients presenting with potential suicidal behaviour. The predictive factors include known risks for suicide, but also other information relating to general health and health service utilisation.

Peer Review reports

Background

Suicidal ideation occurs in more than 10% of the population during their lifetime [1]. Each year, 2% of the population contemplate suicide and 0.3% attempt suicide [2]. Across all age groups, the incidence and prevalence of suicidal behaviour have increased considerably over the past two decades. Suicide is now a more common cause of death than motor vehicle accidents [3–5]. Suicide does not happen without warning. People frequently make contact with health services in the months leading up to their suicide attempt. There is a recognised need to identify those at risk [6, 7] for targeted interventions to stop suicide before it happens [8].

Risk factors for suicide are well-documented, but the list is long. These include: age, male gender, chosen method of attempted suicide, and number of previous attempts; [9] psychiatric diagnoses including anxiety and depression, psychosis, and bipolar disorder; [10–12] social isolation; [13] and potential lethality of previous attempts [14]. Despite the effort to combine these risk factors into risk scores and algorithms to predict suicide risk, [15–17] the predictions are often too poor to be clinically useful [18, 19].

Alternative to a constant set of risk scores that we hope would work for each individual, a broader source of information is available: the comprehensive electronic medical record (EMR). The time-stamped administrative, clinical, and investigative data in the EMR includes information on known risk factors for suicide. Additionally, it contains information on health service encounters that might not be directly related to suicidal behaviour. It is known that patients who attempt suicide frequently attend the emergency department (ED) in the months before their suicide attempt [20, 21]. Many of these attendances are not directly related mental health problems or self-harm. Also, many suicidal patients attend primary care facilities in the months before their suicide attempt, again commonly for reasons not directly related to their psychological distress [22]. Up to 85% of suicidal patients attend hospital outpatients in the twelve months before their suicide attempt, [23] again often for reasons not directly related to psychological morbidity. There is a high prevalence of coexistent physical illnesses amongst patients who exhibit suicidal behaviour [24]. All these contextual factors may contribute to suicide risk, but for each individual patient, they are difficult to be assessed objectively and consistently without a proper tool.

We develop a statistical risk stratification model based on EMR data. The model results from examining suicide attempts and completed suicide in a large cohort of patients who underwent suicide assessment in a regional health service. We compare EMR-based predictions of high-risk suicidal behaviour with clinician predictions, which are based on an 18-point assessment instrument.

Methods

Study design and population

This was a retrospective study using electronic records of inpatient admissions and emergency department (ED) visits within Barwon Health, a regional health service in Australia. As the only tertiary hospital in Greater Geelong, and a catchment area with over 350,000 residents, the hospital’s patient database provides a single access point for information on hospitalisations, ED visits, and medications. Although different IT systems are used by hospitals, EMRs often share similar underlying logical structure. All the diagnoses are often coded using the data standard ICD-10. For generalisability of the model, we focus on EMR data that are either generally available in mental health services or coded using the ICD-10 disease classification.

Patients were included in the study if they were aged 10 years or over, and had received at least one suicide risk assessment between April 2009 and March 2012. A cohort of 7,399 patients (with 16,858 risk assessments) met the inclusion criteria. Risk assessments were carried out either in Barwon Health following presentation to ED or another department or in one of five community health centres. Patient outcomes were observed for a 180 day period following a risk assessment and details of repeat suicide attempts or completed suicides were recorded. A risk assessment was considered a clinically relevant time point for prediction, as care models are devised according to patient risk at that time. If a patient had more than one assessments, one of them would be randomly selected.

Patient records prior to the selected assessment date were used to construct the independent variables. The same data were also available to clinicians during risk assessments. Suicide attempts, of varying lethality, in a 180 day period following the assessment were treated as the dependent variable. The true outcome, for a given period, is the actual risk class determined from patient records for that period. As outlined before, direct modelling of suicide risk is difficult because of low base rate of suicides. Modelling risk of suicide attempts, a strong predictor of suicide, [25] is a pragmatic measure to circumvent this difficulty. Moreover, it allowed us to define a set of risk classes for the suicide attempts of varying lethality levels, consistent with previous published work [26].

Self-harm events are recorded in the hospital database using ICD-10 codes. For example, an ED visit due to by attempted hanging, would be assigned the ICD-10 code X70. We exclude those events which may result from accidents or assaults (instead of intentional self-harm). In Australia, such events are coded with the following ICD-codes: Transport Accidents (V01-V99, Y85); Falls (W00-W19); Accidental Poisoning (X40-X49); Assault (X85-Y09,Y87.1).

For each stipulated period following a risk assessment, the risk class was defined as follows:

  1. 1.

    The period has high-risk if, during the period, the patient had an ED visit or inpatient admission with a high-lethality diagnosis (Table  1).

Table 1 ICD-10 codes identified to correlate with moderate or high lethality events
  1. 2.

    The period has moderate-risk if, during the period, the patient had an ED visit or inpatient admission with a moderate-lethality diagnosis (Table  1).

  2. 3.

    Otherwise, the period has a low-risk (patients with low lethality suicide attempts or no suicide attempts).

Predictive algorithms were constructed from the derivation cohort to predict risk in different post-assessment periods—30, 60, 90, and 180 days. The predictive model takes demographic and historical data prior to a risk assessment, and identifies the post risk-assessment class, for a desired period.

Ethics approval was obtained from the Hospital and Research Ethics Committee at Barwon Health, with whom Deakin University has reciprocal ethics authorisation (approval number 12/83).

Clinician risk assessment

The clinicians’ prediction of risk involves using a risk assessment checklist. Clinical protocols at Barwon Health require a suicide risk assessment to be completed on intake to care, every 91 days during an episode of care and on discharge. For the purpose of this study all available suicide risk assessment data was extracted from the data warehouse. The primary purpose of the assessment is to ensure that consideration of suicide risk is medico-legally documented. Under the Victorian Mental Health Act, mental health services are required to report deaths of patients who are either currently receiving care or have had contact with the service in the past six months. Patient deaths are reported through a centralised risk management system, which is also reconciled against death certificates and a registry of deaths. Deaths reported to the coroner are also tracked to the point where the coroner determines whether the death was due to suicide. All suicides and other unnatural deaths are subject to a comprehensive case review. Data on self-harm was identified by diagnostic codes for intentional overdoses and self-inflicted injuries from admission and emergency presentation coding. The suicide risk assessment instrument used was developed by Barwon Health in 1999, based on known risk factors in the literature. It has been in use for more than 13 years as no alternative risk scores have been shown to be more effective. One of the purposes of this study was to validate use of this risk assessment tool. The tool consists of 18 items, each graded on a 3-point scale (low/moderate/high) covering suicidal ideation, suicide plan, access to means, prior attempts, anger/hostility/impulsivity, depression (current level), anxiety, disorientation/disorganisation, hopelessness, identifiable stressors, substance abuse, psychosis, medical status, withdrawal from others, expressed communication, psychiatric service history, coping strategies and supportive others (connectedness).

Derived predictor variables

Patient specific information—such as interactions with health services, ICD-10 diagnostic codes (mental disorders and other comorbidities), diagnosis-related groups (DRGs) and procedures, demography, and medications—were used to derive predictors. Diagnostic codes were used to derive following subsets: injuries classified as moderate or high risk, mental diagnoses mapped into Australia’s Mental Health Diagnostic Groups, and relevant codes mapped into Elixhauser comorbidities [27]. To ensure generalisability, medications were mapped according to the World Health Organisation’s Anatomical Therapeutic Chemical (ATC) classification system. Different from most existing clinical predictive models, derived predictors are specific in the temporal dimension, encoding the knowledge that prior events may influence future risks at multiple time scales. More specifically, occurrences of historical data were aggregated and normalised over several time periods before the assessment point: 0–3 months, 3–6 months, 6–12 months, 12–24 months, and 24–48 months. Overall 5,322 variables were derived, but only those that appeared more than 4 times among the patients at risk in the derivation cohort were kept, resulting in 202 variables. For robustness, variable values outside the 1st and the 99th percentiles were considered outliers and rounded to the nearest percentiles.

Derivation of the risk stratification model

For predictive modelling, we adopted a recently established statistical technique L1-penalised continuation-ratio model for ordinal outcomes [28]. Different from traditional ordinal regression models that assume monotone effects of predictor variables [29], the continuation-ratio model uses separate coefficients for different outcome classes, at the same time automatically selects highly predictive variables (and their time scales) for each class. As prior studies suggest that predictors of the two classes may be different [30], it is desirable to model the moderate-risk and the high-risk classes using separate coefficients. The variable selection capability comes from the lasso-like shrinkage [31] that forces variables with weak association with the outcomes to have zero coefficients. The shrinkage strength was adjusted so that the predictive performances on the derivation and validation cohorts were similar, and thus ensuring no over-optimism in the estimated model [32].

A single prediction risk score was computed, representing the expected lethality level. The risk score takes a value from 0 to 2, with 0, 1, and 2 representing low, moderate, and high risk respectively. This score results in a simple decision rule: Given two thresholds in the range 0 – 2, the outcome class is low-risk if the score is less than the lower threshold, moderate-risk if the score is below the upper threshold and high-risk otherwise. The thresholds can be adjusted according to the clinical need.

The contribution of individual risk factors toward each risk class was assessed using bootstrap. That is, the models were estimated multiple times from bootstrap samples, and variable coefficients were then collected and statistics were derived. Among them are Wald statistics, confidence intervals, variable importance, and selection probability. Variable importance is defined as the product of absolute mean coefficient and variable standard deviation. Selection probability is the chance that a variable is being selected by applying the lasso shrinkage.

Model validation

To assess the performance of prediction methods, the 7,399 patients were randomly divided into a derivation cohort of 4,911 patients and a validation cohort of 2,488 patients. The risk score assigned by the predictive model and clinician-assigned overall risk were validated against true outcomes on the validation cohort. The risk score makes it simple for estimating performance measures for multiple binary decisions such as high-risk versus the rest, or moderate/high-risk versus low-risk. We reported here Area under the ROC Curves (AUC), sensitivity and specificity. The confidence intervals for these measures were estimated using bootstrap.

Results

Patients

Characteristics of patients included in the study are summarised in Table 2. In the follow up periods of 30 days, 90 days, and 180 days, there were 7, 9, and 13 suicides. The mean age was 41.2 years (11 – 101 years). Overall, 2,479 (33.5%) patients had changed postcode in the preceding 12 months. 3,953 (53.2%) attended ED in the 3 months before suicide assessment, and 4,436 (60.0%) had at least one hospital admission during the preceding 12 months. Of the 7,399 patients, 1,562 (21.1%) had a moderate lethality suicide attempt in the 12 months preceding their assessment and 646 (8.7%) patients had a high lethality attempt. A further breakdown of factors according to the outcomes within 90-day is presented in Table 3. Overall, risk factors were more prevalent in the more risky groups. The prevalence of males, pensioners, prior moderate-risk events, past 3-month admission and alcohol abuse were highest in the moderate-risk group. Other risk factors were most prevalent in the high-risk group.

Table 2 Demographic and clinical characteristics
Table 3 Risk factors within outcome groups (low, moderate and high-risk events) in 90-days

Validation of the 18-point risk score

Risk scores from the 18-point risk assessment instrument with respect to 90-day outcomes are shown in Table 4. Overall most identified factors were scored higher in the high risk group. Not surprisingly, the differential between low and high risk groups was highest for the factor ‘prior attempt’ (the average score for the high risk group was 1.70 compared to 1.26 for the low risk).

Table 4 Average risk scores assigned to 18-item checklist within true outcome groups (low, moderate and high-risk events) in 90-days

Comparison of clinician and machine model predictions

To assess the performance of the predictive model using EMR data, we compared it with the clinical assessment (prompted by the 18-point check list). The discriminative performance of these two methods, in terms of area under the ROC curve (AUC), is presented in Table 5. Prediction was compared over 30, 60, 90, and 180 days. The clinician prediction had relatively low predictive ability with AUCs of 0.55 to 0.59 over the four time points when predicting high-risk events. AUCs for the EMR model were consistently better, ranging from 0.73 to 0.79. Similar differentials were also observed when predicting either moderate or risk events, where the AUCs were in the range 0.52 - 0.54 for clinicians and 0.71 - 0.79 for EMR models.

Table 5 Area under the ROC curve (AUC, 95% CIs) of clinicians versus EMR-based model on the validation cohort

In identifying high risks within 90 days, clinicians achieved sensitivity of 0.08 (95% CIs: 0.02-0.2) at specificity of 0.97 (95% CIs: 0.96-0.98). The EMR model, on the other hand, reached sensitivity of 0.28 (95% CIs: 0.16-0.41) at the same specificity. Note that this low sensitivity was due to unusually high specificity of 0.97. At specificity of 0.72 (95% CIs: 0.70-0.73) the EMR model had sensitivity of 0.70 (95% CIs: 0.56-0.83). For either moderate or high-risks, clinicians scored 0.27 (95% CIs: 0.19-0.35) sensitivity and 0.82 (95% CIs: 0.81-0.84) specificity. The scores for the EMR model were higher at specificity of 0.58 (95% CIs: 0.50-0.66) at the same specificity.

Of two patients in the validation cohort who completed suicide during 30 days of follow-up, neither was identified as high-risk by clinicians while one was categorised as high-risk by the EMR model. Three patients completed suicide within 90 days. None were identified as high-risk by clinicians while the EMR model correctly classified two of the three patients.

Risk factors

Table 6 shows the relative importance of the top factors that are predictive of high-risk events in 90 days following suicide assessment. Prior high-risk events, injuries and poisoning were indicative of subsequent high-risk events. Mental health male patients who moved homes (as approximated by postcode change) were also at risk. These risk factors were related but different from those associated with subsequent moderate-risk events (Table 7). For example, prior emergency visits, addictive drugs treatment and moderate-risk events were predictive of the future moderate-risk but not the high-risk.

Table 6 High-risk predictive factors in the EMR model for 3-month prediction
Table 7 Moderate-risk predictive factors in the EMR model for 3-month prediction

Discussion

We have developed a robust predictive model that takes information rich administrative EMR data and stratifies mental health a patient into three ordinal categories: low, moderate and high-risks of suicidal behaviours in 30, 60, 90 and 180 days following a risk assessment. For comparison against the common practice of assessment by clinicians, we validated an 18-point checklist developed and used at Barwon Health. The EMR model showed much greater predictive ability than clinicians using the checklist in identifying high-risk patients.

We followed 7,399 consecutive patients judged to be at risk of suicide for 180 days following a clinical assessment. Although this time frame is considerably short for rare events such as suicides, this was chosen for practical purposes in treatment and resources planning. Our first aim was to document risk factors associated with high and moderate lethality behaviours. We have confirmed that high lethality suicidal behaviour is associated with social and demographic factors and prior high-risk events of injuries and poisoning. On the other hand, moderate lethality behaviour is related to prior interactions with health services, history of moderate-risk events, mental disorders and lifestyle problems. These results are in line with previous studies, in that the number and potential lethality of previous suicide attempts predict lethality of future attempts [9, 14]. Also, a range of mental health diagnoses were, as previously shown, predictive of future attempts during follow-up [10–12]. Social factors such as postcode change are strongly associated with suicide risk, concordant with recent findings [13, 33]. We document that 66.8% of patients undergoing suicide assessment attended ED and that 41.8% of patients had been admitted to hospital in the year before their assessment. The majority of these encounters were not for episodes of self-harm or even for mental health problems. This information added stimulus to exploring use of administrative EMR data in risk stratification.

Our second aim was to examine the contribution of mental and physical health diagnoses to suicide risk. It appears that mental health diagnoses are powerful predictors of moderate-risk behaviour. However, medical conditions also contribute to risk. Previous studies have confirmed that hospital admission for non-psychiatric conditions are associated with increased suicide risk [23, 24].

The next aim was to validate the 18-point risk checklist. Most of the factors in the checklist were more prevalent in patients with higher risk. To verify that if the effectiveness of checklist could be improved further, we tested same statistical framework in this paper on predictors derived from the 18 aspects and demographic data. The results were very encouraging – the algorithm outperformed the clinician’s overall risk assignment, and the difference was statistically significant. We conclude, therefore, that the risk checklist is a valid decision support tool. Although there has been some pessimism about the ability to predict suicide risk in the clinical setting, [18, 19] recent reports of predictive models suggest that clinically useful risk stratification is possible using recognised risk factors [15–17].

Our fourth aim was to document the predictive performance of clinicians prompted by the 18-point risk checklist. This was relatively poor, underlining the potential utility of a decision support tool. A recent study documented the potential for clinical decision rules to augment clinical performance in risk stratification of patients who might be at risk of suicide [34]. The models studied had high sensitivity but low specificity, unlike those reported in our study which had lower sensitivity but high specificity. Recent systemic studies of the factors associated with high risk in suicide-prone patients have added to knowledge on the potential contribution of known risk factors to prognostic stratification, and may help in the design of effective decision support algorithms [35, 36].

Our final aim was to use predictive models applied to data from the EMR to derive predictive models for high-risk suicide behaviour. Models for prediction of high-risk behaviour used administrative, social and demographic, and clinical data. Risk stratification using these models was more accurate than clinical stratification. The potential of EMR data to identify patients at high-risk of suicide from overdose has been documented in a recent study of paediatric patients [37].

The strengths of this study include the large size of the dataset, the fact that patients were consecutively recruited from a defined geographical region, and the comprehensive follow-up. Suicide assessments in both hospital and community settings were included. The models based on EMR could be updated in real-time, and make use of data that are routinely collected. The predictors derived from the EMR data were standardised, and thus the tools can be generalisable to sites with similar EMR systems. Limitations are that it is a retrospective study confined to a single location. The predictive models were validated in the setting in which they were derived but their performance may differ in other settings. The problem studied is inherently difficult to predict and there may be limits to the performance of predictive models due to the fact that individual factors are only weakly predictive.

Conclusions

In conclusion, demographic, social, mental and physical health, and administrative data related to health service encounters can all be used to identify patients at risk of self-harm behaviour. A history of high risk events are indicative of short-term future high-lethality suicide attempts. Clinicians’ assessment could be enhanced by using data from a simple checklist of known risk factors. Modern statistical techniques applied to complex data from a comprehensive EMR has considerable potential to improve risk stratification of patients that may engage in repeated self-harm attempts.

Abbreviations

AUC:

Area under the ROC Curve

CI:

Confidence interval

ED:

Emergency department

EMR:

Electronic medical record

ICD-10:

International statistical classification of diseases and related health problems 10th revision

ROC:

Receiver operating characteristic.

References

  1. Nock MK, Green JG, Hwang I, McLaughlin KA, Sampson NA, Zaslavsky AM, Kessler RC: Prevalence, correlates, and treatment of lifetime suicidal behavior among adolescents: results from the national comorbidity survey replication adolescent supplement. JAMA Psychiatry. 2013, 70 (3): 300-310. 10.1001/2013.jamapsychiatry.55.

    Article  PubMed  Google Scholar 

  2. Borges G, Nock MK, Haro Abad JM, Hwang I, Sampson NA, Alonso J, Andrade LH, Angermeyer MC, Beautrais A, Bromet E, Bruffaerts R, de Girolamo G, Florescu S, Gureje O, Hu C, Karam EG, Kovess-Masfety V, Lee S, Levinson D, Medina-Mora ME, Ormel J, Posada-Villa J, Sagar R, Tomov T, Uda H, Williams DR, Kessler RC: Twelve-month prevalence of and risk factors for suicide attempts in the world health organization world mental health surveys. J Clin Psychiatry. 2010, 71 (12): 1617-1628. 10.4088/JCP.08m04967blu.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Spiller HA, Appana S, Brock GN: Epidemiological trends of suicide and attempted suicide by poisoning in the US: 2000–2008. Legal Med. 2010, 12 (4): 177-183. 10.1016/j.legalmed.2010.04.005.

    Article  PubMed  Google Scholar 

  4. Ting SA, Sullivan AF, Boudreaux ED, Miller I, Camargo CA: Trends in US emergency department visits for attempted suicide and self-inflicted injury, 1993–2008. Gen Hosp Psychiatry. 2012, 34 (5): 557-565. 10.1016/j.genhosppsych.2012.03.020.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Centers for Disease C, Prevention: Suicide among adults aged 35–64 years - United States, 1999–2010. MMWR Morb Mortal Wkly Rep. 2013, 62 (17): 321-325.

    Google Scholar 

  6. Huffman LC, Wang NE, Saynina O, Wren FJ, Wise PH, Horwitz SM: Predictors of hospitalization after an emergency department visit for California youths with psychiatric disorders. Psychiatr Serv. 2012, 63 (9): 896-905.

    Article  PubMed  Google Scholar 

  7. Allen MH, Abar BW, McCormick M, Barnes DH, Haukoos J, Garmel GM, Boudreaux ED: Screening for suicidal ideation and attempts among emergency department medical patients: instrument and results from the psychiatric emergency research collaboration. Suicide Life Threat Behav. 2013, 43 (3): 313-323. 10.1111/sltb.12018.

    Article  PubMed  Google Scholar 

  8. Choi JW, Park S, Yi KK, Hong JP: Suicide mortality of suicide attempt patients discharged from emergency room, nonsuicidal psychiatric patients discharged from emergency room, admitted suicide attempt patients, and admitted nonsuicidal psychiatric patients. Suicide Life Threat Behav. 2012, 42 (3): 235-243. 10.1111/j.1943-278X.2012.00085.x.

    Article  PubMed  Google Scholar 

  9. Perry IJ, Corcoran P, Fitzgerald AP, Keeley HS, Reulbach U, Arensman E: The incidence and repetition of hospital-treated deliberate self harm: findings from the world’s first national registry. PLoS ONE [Electronic Resource]. 2012, 7 (2): e31663-10.1371/journal.pone.0031663.

    Article  CAS  Google Scholar 

  10. Gonda X, Fountoulakis KN, Kaprinis G, Rihmer Z: Prediction and prevention of suicide in patients with unipolar depression and anxiety. Ann Gen Psychiatry. 2007, 6: 23-10.1186/1744-859X-6-23.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Gomez-Duran EL, Martin-Fumado C, Hurtado-Ruiz G: Clinical and epidemiological aspects of suicide in patients with schizophrenia. Actas Esp Psiquiatr. 2012, 40 (6): 333-345.

    PubMed  Google Scholar 

  12. Gonda X, Pompili M, Serafini G, Montebovi F, Campi S, Dome P, Duleba T, Girardi P, Rihmer Z: Suicidal behavior in bipolar disorder: epidemiology, characteristics and major risk factors. J Affect Disord. 2012, 143 (1–3): 16-26.

    Article  PubMed  Google Scholar 

  13. Haw C, Hawton K: Living alone and deliberate self-harm: a case–control study of characteristics and risk factors. Soc Psychiatry Psychiatr Epidemiol. 2011, 46 (11): 1115-1125. 10.1007/s00127-010-0278-z.

    Article  PubMed  Google Scholar 

  14. Sapyta J, Goldston DB, Erkanli A, Daniel SS, Heilbron N, Mayfield A, Treadway SL: Evaluating the predictive validity of suicidal intent and medical lethality in youth. J Consult Clin Psychol. 2012, 80 (2): 222-231.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Waern M, Sjostrom N, Marlow T, Hetta J: Does the suicide assessment scale predict risk of repetition? A prospective study of suicide attempters at a hospital emergency department. Eur Psychiatry: the J Assoc Eur Psychiatrists. 2010, 25 (7): 421-426. 10.1016/j.eurpsy.2010.03.014.

    Article  CAS  Google Scholar 

  16. Fountoulakis KN, Pantoula E, Siamouli M, Moutou K, Gonda X, Rihmer Z, Iacovides A, Akiskal H: Development of the risk assessment suicidality scale (RASS): a population-based study. J Affect Disord. 2012, 138 (3): 449-457. 10.1016/j.jad.2011.12.045.

    Article  PubMed  Google Scholar 

  17. Stefansson J, Nordstrom P, Jokinen J: Suicide intent scale in the prediction of suicide. J Affect Disord. 2012, 136 (1–2): 167-171.

    Article  CAS  PubMed  Google Scholar 

  18. Bolton JM, Spiwak R, Sareen J: Predicting suicide attempts with the SAD PERSONS scale: a longitudinal analysis. J Clin Psychiatry. 2012, 73 (6): e735-e741. 10.4088/JCP.11m07362.

    Article  PubMed  Google Scholar 

  19. Ryan CJ, Large MM: Suicide risk assessment: where are we now?. Med J Aust. 2013, 198 (9): 462-463. 10.5694/mja13.10437.

    Article  PubMed  Google Scholar 

  20. Sakinofsky I: Attendance at accident and emergency for deliberate self harm predicts increased risk of suicide, especially in women. Evid Based Ment Health. 2005, 8 (4): 97-10.1136/ebmh.8.4.97.

    Article  PubMed  Google Scholar 

  21. Da Cruz D, Pearson A, Saini P, Miles C, While D, Swinson N, Williams A, Shaw J, Appleby L, Kapur N: Emergency department contact prior to suicide in mental health patients. Emerg Med J. 2011, 28 (6): 467-471. 10.1136/emj.2009.081869.

    Article  CAS  PubMed  Google Scholar 

  22. Luoma JB, Martin CE, Pearson JL: Contact with mental health and primary care providers before suicide: a review of the evidence. Am J Psychiatr. 2002, 159 (6): 909-916. 10.1176/appi.ajp.159.6.909.

    Article  PubMed  Google Scholar 

  23. Liu HL, Chen LH, Huang SM: Outpatient health care utilization of suicide decedents in their last year of life. Suicide Life Threat Behav. 2012, 42 (4): 445-452. 10.1111/j.1943-278X.2012.00103.x.

    Article  PubMed  Google Scholar 

  24. Qin P, Webb R, Kapur N, Sorensen HT: Hospitalization for physical illness and risk of subsequent suicide: a population study. J Intern Med. 2013, 273 (1): 48-58. 10.1111/j.1365-2796.2012.02572.x.

    Article  CAS  PubMed  Google Scholar 

  25. Galfalvy HC, Oquendo MA, Mann JJ: Evaluation of clinical prognostic models for suicide attempts after a major depressive episode. Acta Psychiatr Scand. 2008, 117 (4): 244-252. 10.1111/j.1600-0447.2008.01162.x.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. Elliott AJ, Pages KP, Russo J, Wilson LG: A profile of medically serious suicide attempts. J Clin Psychiatry. 1996, 57 (12): 567-571. 10.4088/JCP.v57n1202.

    Article  CAS  PubMed  Google Scholar 

  27. Elixhauser A, Steiner C, Harris DR, Coffey RM: Comorbidity measures for use with administrative data. Med Care. 1998, 36 (1): 8-27. 10.1097/00005650-199801000-00004.

    Article  CAS  PubMed  Google Scholar 

  28. Archer KJ, Williams AA: L1 penalized continuation ratio models for ordinal response prediction using high-dimensional datasets. Stat Med. 2012, 31 (14): 1464-1474. 10.1002/sim.4484. doi: 10.1002/sim.4484 [published Online First: Epub Date]

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Ananth CV, Kleinbaum DG: Regression models for ordinal responses: a review of methods and applications. Int J Epidemiol. 1997, 26 (6): 1323-1333. 10.1093/ije/26.6.1323.

    Article  CAS  PubMed  Google Scholar 

  30. Fowler JC, Piers C, Hilsenroth MJ, Holdwick DJ, Padawer JR: The rorschach suicide constellation: assessing various degrees of lethality. J Pers Assess. 2001, 76 (2): 333-351. 10.1207/S15327752JPA7602_13. doi: 10.1207/S15327752JPA7602_13[published Online First: Epub Date]

    Article  CAS  PubMed  Google Scholar 

  31. Tibshirani R: Regression shrinkage and selection via the Lasso. J Roy Stat Soc B Met. 1996, 58 (1): 267-288.

    Google Scholar 

  32. Moons KG, Donders AR, Steyerberg EW, Harrell FE: Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example. J Clin Epidemiol. 2004, 57 (12): 1262-1270. 10.1016/j.jclinepi.2004.01.020. doi: 10.1016/j.jclinepi.2004.01.020 [published Online First: Epub Date]

    Article  CAS  PubMed  Google Scholar 

  33. Ishii N, Terao T, Araki Y, Kohno K, Mizokami Y, Arasaki M, Iwata N: Risk factors for suicide in Japan: a model of predicting suicide in 2008 by risk factors of 2007. J Affect Disord. 2013, 147 (1–3): 352-354.

    Article  PubMed  Google Scholar 

  34. Bilen K, Ponzer S, Ottosson C, Castren M, Owe-Larsson B, Ekdahl K, Pettersson H: Can repetition of deliberate self-harm be predicted? A prospective multicenter study validating clinical decision rules. J Affect Disord. 2013, 149 (1–3): 253-258.

    Article  PubMed  Google Scholar 

  35. Challis S, Nielssen O, Harris A, Large M: Systematic meta-analysis of the risk factors for deliberate self-harm before and after treatment for first-episode psychosis. Acta Psychiatr Scand. 2013, 127 (6): 442-454. 10.1111/acps.12074.

    Article  CAS  PubMed  Google Scholar 

  36. Hawton K, Casanas ICC, Haw C, Saunders K: Risk factors for suicide in individuals with depression: a systematic review. J Affect Disord. 2013, 147 (1–3): 17-28.

    Article  PubMed  Google Scholar 

  37. de Achaval S, Feudtner C, Palla S, Suarez-Almazor ME: Validation of ICD-9-CM codes for identification of acetaminophen-related emergency department visits in a large pediatric hospital. BMC Health Serv Res. 2013, 13: 72-10.1186/1472-6963-13-72.

    Article  PubMed  PubMed Central  Google Scholar 

Pre-publication history

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Svetha Venkatesh.

Additional information

Competing interests

MB has received Grant/Research Support from the NIH, Cooperative Research Centre, Simons Autism Foundation, Cancer Council of Victoria, Stanley Medical Research Foundation, MBF, NHMRC, Beyond Blue, Rotary Health, Geelong Medical Research Foundation, Bristol Myers Squibb, Eli Lilly, Glaxo SmithKline, Meat and Livestock Board, Organon, Novartis, Mayne Pharma, Servier and Woolworths, has been a speaker for Astra Zeneca, Bristol Myers Squibb, Eli Lilly, Glaxo SmithKline, Janssen Cilag, Lundbeck, Merck, Pfizer, Sanofi Synthelabo, Servier, Solvay and Wyeth, and served as a consultant to Astra Zeneca, Bristol Myers Squibb, Eli Lilly, Glaxo SmithKline, Janssen Cilag, Lundbeck Merck and Servier.

Authors’ contributions

TT implemented the prediction model. RH and SV conceived of the study. WL performed the data extraction. TT, WL, DP, and SV carried out the statistical analysis. RH, MB, RK and SV participated in coordination. All authors participated in the design of the study, helped to draft the manuscript. All authors read and approved the final manuscript.

Rights and permissions

Open Access  This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/.

The Creative Commons Public Domain Dedication waiver (https://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article

Tran, T., Luo, W., Phung, D. et al. Risk stratification using data from electronic medical records better predicts suicide risks than clinician assessments. BMC Psychiatry 14, 76 (2014). https://doi.org/10.1186/1471-244X-14-76

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1471-244X-14-76

Keywords