Abstract
The present study reports on the development of a measure designed to assess satisfaction with service among civil and forensic psychiatric inpatients. Survey items drawn from a review of existing measures were administered to 427 forensic and 416 civil male psychiatric inpatients. In an effort to develop a reliable and valid measure, a rigorous test development procedure was undertaken involving item and principal components analyses followed by a confirmatory factor analysis of the remaining items. For forensic inpatients, a 14-item questionnaire (Forensic Inpatient Satisfaction Questionnaire; F-ISQ) emerged that addressed four domains of satisfaction: Medication and Treatment, Physical Environment, Telephone Access, and Unit Rules and Procedures. For civil inpatients, an 11-item questionnaire (Civil Inpatient Satisfaction Questionnaire; C-ISQ) included two domains of satisfaction: Needs and Opportunities and Food and Comfort. Strong internal reliability and concurrent validity with other established measures of patient satisfaction were demonstrated.
Proper measurement of psychiatric patient satisfaction is essential in the evaluation of mental health services. Levels of patient satisfaction are considered to be an indicator of quality of care1 and are specifically related to compliance with treatment,2 improvement in health status,3 and future use of psychiatric services.2 Understanding patients' satisfaction with services helps to inform the development of improved mental health programs designed to meet patients' needs. Although in recent decades there has been an increase in the use of satisfaction surveys in the mental health field, it is still uncommon to find patient satisfaction surveys employed on involuntary civil or forensic psychiatry units. Furthermore, despite the increase in measuring patient satisfaction, many surveys are poorly designed and have weak psychometric properties. The purpose of the present study was to report on the development and psychometric properties of an innovative patient satisfaction survey designed for an inpatient psychiatric service to address the needs of civil and forensic inpatient psychiatric populations.
Satisfaction Among Psychiatric Inpatients
There are divergent views regarding the value of obtaining opinions from patients who receive psychiatric services. Some have argued that reports from such patients are unreliable due to the patients' psychiatric conditions,3 while others have posited that social desirability may inflate ratings of satisfaction.4 Indeed, these concerns affect the validity of data collected from patient satisfaction surveys. Despite these criticisms, the usefulness of patient satisfaction in understanding the effectiveness of psychiatric service provisions has been documented.2,5,6
Past research has mostly found that patients are generally satisfied with services offered.3,7 Forensic psychiatric patients also report high levels of satisfaction.8–10 Patients report being most satisfied with the staff (especially the care that they provide and their kindness), as well as the cleanliness of the facility.3,11 In one study, forensic patients tended to be most satisfied with the attitudes of the nursing staff and occupational therapists compared with their doctors, social workers, and psychologists.10
The trend toward satisfied patients is not universal. A psychiatric inpatient unit in South London reported that most patients were actually dissatisfied with their treatment experiences.11 Most of the patients involved in this study were specifically dissatisfied with the number of sessions that they had had with their doctors, and they perceived the involvement of relatives in treatment to be insufficient. It should be noted, however, that most of these patients were admitted voluntarily and carried nonpsychotic diagnoses. Even though many patients surveyed were generally satisfied, most reported that improvements were needed in the domains of food choice, individual patient's level of involvement in treatment decisions, availability of discharge plans, choice of psychiatrist, procedures that manage the side effects of prescribed medication, privacy, and ambient noise.8–12
Measures of Patient Satisfaction
Many surveys that have been designed to assess patient satisfaction were developed pragmatically for a specific facility's use and were not based on established measures of patient satisfaction. For example, in an effort to measure patient satisfaction with antipsychotic medication treatment, Gray and colleagues12 gathered input from clinicians, patients, and researchers to help draft their questionnaire. This survey was then pilot tested on five patients, and their feedback was used to create the final survey. As is true of many surveys, no reliability or validity information was provided. In another example, Wykes and Carroll3 briefly discussed the development of their questionnaire, but did not provide the reader with any information regarding the psychometric properties associated with the instrument. Some studies failed to report in sufficient detail how the measure was developed,13,14 calling into question the validity of reported findings. In particular, valid measures of patient satisfaction in forensic psychiatric units have been lacking. Researchers either modify existing measures to suit their populations (choosing to omit certain questions and to change the wording of questions on Wykes and Carroll's 1993 Maudsley Inpatient Satisfaction Schedule10) or neglect to describe the measure used to assess satisfaction (simply stating that patients were administered an anonymous questionnaire to measure opinions on different forms of treatment14). Exceptions include the Psychiatric Case Satisfaction Questionnaire (PCSQ) developed by Barker and Orrell15 and the Inpatient Evaluation of Service Questionnaire (IESQ) of Meehan et al.16
The PCSQ was developed by psychiatrists for use on an inpatient unit. Barker and Orrell15 report on its strong validity (face, consensual, content, concurrent, and criterion) and reliability (internal consistency and test-retest). They conclude that it is a practical and adequate measure of patient satisfaction in a hospital setting and is considered a general indicator of satisfaction. The IESQ16 underwent exhaustive development, relying on inpatient focus group discussions to generate items, followed by a rating of the importance of each item and the drafting of the questionnaire. Response variability and the factor structure of the IESQ were systematically analyzed. It was concluded that the IESQ was a user-friendly measure of patient satisfaction for psychiatric inpatients. Both the PCSQ and IESQ are relatively brief measures that are designed to assess patient satisfaction in an inpatient setting (18 and 20 items, respectively). It is argued that clinicians are more likely to employ brief, psychometrically sound measures in such settings, rather than to use lengthy measures, despite adequate reliability and validity.
Although the PCSQ and IESQ underwent rigorous development and testing, two limitations are noted. First, to our knowledge, these measures have not undergone extensive test development procedures including a principal components analysis (PCA) of the survey items followed by a confirmatory factor analysis (CFA) in a separate sample from the same population. Such procedures are useful for several reasons. The PCA is used to identify underlying factors that explain responses to questionnaires.17 In other words, rather than simply stating that a group of items on a questionnaire reflects a hypothesized construct, the PCA statistically identifies the constructs and their respective items. CFA allows researchers to evaluate statistically whether the underlying constructs of a questionnaire actually replicate in similar (or different) samples.17 Second, the PCSQ and IESQ were designed with psychiatric inpatient samples, and their use in forensic (versus civil) populations may be insufficient. Satisfaction surveys must be designed for the particular service (i.e., psychiatry) and setting (i.e., forensic inpatient) that is being evaluated. Furthermore, research designed to develop and evaluate the psychometric properties of satisfaction surveys should employ reasonably large sample sizes and minimize potential situations that impact satisfaction ratings (e.g., hospital staff reading satisfaction survey items to patients).18 Accordingly, the purpose of the present study was to develop and investigate the psychometric properties of a survey designed to evaluate patient satisfaction on forensic and civil psychiatric inpatient units. In sum, the present study addresses gaps in the literature on psychiatric patients' satisfaction with care in several ways: conducting rigorous survey development and testing procedures to create a valid and reliable measure of patient satisfaction, designing psychometrically strong satisfaction surveys that can be used for civil and forensic psychiatric patients, examining a larger sample of patients, and minimizing procedures that may affect patients' ratings of their satisfaction. It is argued that, by addressing these limitations, the resulting questionnaire would be a preferred measure for assessing psychiatric inpatients' perceptions of their care.
Method
Participants
The nonrandom sample of convenience (i.e., participants were approached and asked to participate during specific times of the year) consisted of 843 male adult inpatients from forensic and civil psychiatric units of a major metropolitan hospital located in New York City. Patients were admitted and questionnaires administered to them between 2002 and 2007. Their ages ranged from 18 to 81 (mean 36.77, SD 12.01). Roughly half of the psychiatric patients were housed in the forensic units (n = 427, 50.7%) and the rest were housed in the civil units (n = 416, 49.3%) of the hospital. The forensic units housed all males and tended to have significantly younger patients (t(841) = −6.64, p < .01), as well as a higher proportion of African American patients (χ2 (5, n = 828) = 55.67, p < .01), than did the civil units.
The forensic and civil samples were further divided to conduct both exploratory and confirmatory factor analyses. Cases were randomly assigned to two subsample groups for both forensic (n = 213 for forensic subsample 1 and n = 214 for forensic subsample 2) and civil (n = 208 for civil subsample 1 and n = 208 for civil subsample 2) samples. Subsample 1 from each sample was used for the principal components analyses, and subsample 2 from each sample was used for the confirmatory factor analyses. No significant differences were found between the subsamples and the respective samples. Forensic inpatient subsamples had similar distributions of race/ethnicity (χ2 (5, n = 427) = 6.84, p = .23) and age (t(426) = 0.22, p = .83) at the time of testing, while civil inpatients in the subsample were also similar in distribution of males and females (χ2(1, n = 416) = .01, p = .91) and race/ethnicity (χ2(5,n = 416) = 4.22, p = .52), as well as age (t(414) = 0.80, p = .65).
The forensic inpatient psychiatric service provides mental health services to male inmates from the New York correctional system who require light-to-maximum security coverage. The legal status of patients is pre-arraignment, awaiting trial, or postconviction (serving sentences of one year or less). The civil inpatient units provide mental health services to male and female adult patients who present with a variety of psychiatric difficulties including, but not limited to, major mood disorders, psychoses, and comorbid substance abuse problems. Only male civil inpatients were included in the present study. Admission to the civil unit can be voluntary or involuntary, depending on each patient's presentation. They may be involuntarily committed because they are believed to pose an imminent danger to themselves or others, their behavior is deemed intolerable to the community (e.g., the patient is unable to care adequately for him/herself in the community), or outpatient care has failed (i.e., when symptoms persist despite the patient's compliance with outpatient treatment).
It is important to note that the quality of care and types of services offered to patients on both the inpatient forensic and civil units are comparable. Patients on both units are assigned to a designated multidisciplinary treatment team and have access to medical doctors, psychiatrists, psychologists, social workers, nursing staff, and activity therapists. Thus, the care received on both units is considered similar. The primary difference between the environments of the two units is the increased security on the forensic units (e.g., presence of correctional officers, gates controlled by officers rather than locked doors).
Procedure
The Inpatient Satisfaction Questionnaire (ISQ) was administered by Master's and doctoral-level graduate students (psychology research externs). Before ISQ administration, all research externs underwent individual or group (two to three persons) training with the first author. Training involved three primary goals: preparing the externs for a wide variety of questions and scenarios that may arise within an inpatient population; standardizing the introduction, presentation, and administration of the ISQ; and ensuring that the externs understand and respect patients, focusing on the patient's right to refuse participation at any stage of the administration. Externs were observed during the first administration of the ISQ (and on subsequent administrations if necessary). During their second and third ISQ administrations, they were accompanied by more senior externs.
The ISQ has been administered biannually since 2002 (annually between 2005 and 2007). Collection of patient satisfaction data was initially part of a Department of Psychiatry Performance Improvement Project (PIP); therefore, informed consent was not needed from participants. However, approval from the governing institutional review board (IRB), including a waiver of informed consent, was obtained in 2005, and the study was formally approved. The externs read instructions verbatim to all participants, explicitly stating that the patients' involvement was voluntary and that they had the right to refuse to participate at any time. Participants were told that their responses would remain confidential and would never become part of their hospital record. They were informed that any information discussed with hospital staff or administration would not reveal their identity. Patients hospitalized for at least five days were considered eligible to participate (the minimum time during which the patient must be seen by members of all treatment team disciplines). Before approaching any individual, the externs spoke with the unit nursing staff to determine which patients might not be prepared (at that moment) to take the ISQ, so that they could temporarily screen out floridly psychotic or recently violent patients.
Questionnaires
The Inpatient Satisfaction Questionnaire
The Inpatient Satisfaction Questionnaire (ISQ) was devised in September 2002 to meet the requirements of an upcoming accreditation visit by the Joint Commission for the Accreditation of Hospital Organizations (JCAHO). Patient satisfaction, periodically assessed among civil inpatients, had gone largely unaddressed within the forensic inpatient population, in part because of the realization that assessing a forensic population with a tool devised for civil patients may not provide an accurate reflection of the needs and questions of forensic inpatients. Therefore, a new inpatient satisfaction questionnaire that would satisfy the JCAHO requirement and could also be used as part of a Department of Psychiatry Performance Improvement Project (PIP) was designed.
The selection of items and domains of satisfaction for the ISQ was guided by a review of the relevant literature and several specific research articles.10,11,15,19 Eighty-two items (including 57 general, 13 discipline specific, and 12 open-ended questions) were chosen to assess eight hypothesized areas of satisfaction: physical environment, basic needs, comfort and safety, unit orientation, treatment and medication, therapy, insight, and Department of Correction-related concerns (specifically for the forensic units). All items were distributed to five senior clinical members of the Division of Forensic Psychiatry (e.g., psychiatrists and psychologists) and the Deputy Director of Psychiatry for their review. Members were asked to indicate the items that best represented the hypothesized domains. They were also asked to review the proposed items, keeping in mind the general educational level of the patients and appropriateness to setting. Final items were agreed on by consensus among the forensic staff members and psychiatry administration. The final version of the ISQ contained 59 items comprising three separate sections (General, Specific, and Short-Answer). The General section contained 41 items that assessed general satisfaction. Patients rated the degree to which they agreed with each statement (e.g., “The food is bad here.”) on a five-point Likert-type scale (0, N/A or no opinion; 1, strongly disagree; 2, disagree; 3, agree; and 4, strongly agree). The section on Specifics assessed particular areas of patient satisfaction. Patients indicated their level of agreement with 11 statements (e.g., “[Staff member] has checked on me regularly since I got here.”) using the same five-point Likert-type scale applied to four specific staff groups on each unit (psychiatry, nursing, social work, and activity therapy). The final section of the survey was the Short-Answer portion, which contained seven questions that asked patients to elaborate on their individual experiences on the unit (e.g., “What was the worst thing about your stay?”).
The present study focused only on the 41 items from the general section of the ISQ. These items had a Flesh Reading Ease score of 74.7, which is considered the preferred reading level,20 and a Flesh-Kincaid Grade Level of 4.7, meaning the items should be understood by individuals who have nearly completed the fourth grade. All patients in the present sample completed the ISQ (n = 843). Total and scale scores were computed according to the factor structures tested and are described in the following sections.
Psychiatric Case Satisfaction Questionnaire
The PCSQ15 contains 18 items (e.g., “I am satisfied with the care I get here.”) that assess satisfaction rated on a five-point Likert scale (1, strongly agree; 2, agree; 3, uncertain; 4, disagree; and 5, strongly disagree). In an effort to make comparisons between measures consistent, all scales were reversed for the present study such that higher scores indicated increased satisfaction. A total of 603 (71%) patients completed the PCSQ between 2005 and 2007, including 228 forensic (n = 123 for subsample 1, n = 105 for subsample 2) and 374 civil (n = 186 for subsample 1, n = 188 for subsample 2) patients. The total scores calculated ranged from 19 to 88 (mean 49.56, SD 12.08). Internal consistency was high, with a Cronbach's α of 0.90 (0.87 and 0.89 for forensic and 0.92 and 0.91 for civil subsamples). Forensic (mean 50.48, SD 11.17) and civil (mean 49.03, SD 12.59) inpatients reported similar levels of satisfaction (t(600) = 1.43, p = .15). Scale scores did not differ significantly within samples such that the subsamples in the respective inpatient group had similar PCSQ scores (forensic subsamples: t(226) = 0.33, p = .68 and civil subsamples: t(372) = 0.56, p = .22).
Inpatient Evaluation of Service Questionnaire
The IESQ16 is a 20-item survey in which respondents rate different aspects of their experience on the unit (e.g., “Availability of nursing staff”) on a five-point Likert-type scale (1, poor; 2, fair; 3, good; 4, very good; and 5, excellent). The IESQ includes three scales of patient satisfaction: Staff-Patient Alliance (SPA, 10 items), Satisfaction with Environment (SE, 6 items), and Satisfaction with Treatment (ST, 4 items). Patients admitted between 2005 and 2007 were administered the IESQ; 548 inpatients participated (65%): 206 forensic inpatients (n = 119 and 87 subsamples 1 and 2) and 341 civil inpatients (n = 185 and 156 subsamples 1 and 2). Total and scale scores were computed for the sample. Total IESQ scores ranged from 20 to 100 (mean 58.08, SD 20.25) and did not differ between forensic (mean 56.78, SD 19.71) or civil inpatient (mean 58.80, SD 20.56) groups (t(545) = −1.13, p = .26). SPA scores ranged from 10 to 50 (mean 28.67, SD 10.79) and were similar across patient groups (t(545) = −1.66, p = .10; forensic sample: mean 27.72, SD 10.60; civil sample: mean 29.24, SD 10.88). SE scores ranged from 6 to 30 (mean 17.43, SD 5.88) and were also similar across the forensic (mean 17.16, SD 5.64) and civil patient groups (mean 17.59, SD 6.03), t(545) = −0.87, p = .39). ST scores ranged from 4 to 20 (mean 11.50, SD 4.57) and were significantly higher for civil patients (mean 11.86, SD 4.59) than forensic patients (mean 10.89, SD 4.48), t(545) = −2.55, p = .01. Subsample scores within the samples did not differ for any of the scales, or for the total score.
Internal consistency was high, with a Cronbach's α of 0.96 for the total scale (forensic subsamples: 0.95 and 0.97; civil subsamples: 0.97 and 0.97), 0.94 for SPA (forensic subsamples: 0.93 and 0.95; civil subsamples: 0.94 and 0.94), 0.85 for SE (forensic subsamples: 0.83 and 0.86; civil subsamples: 0.86 and 0.86), and 0.90 for ST (forensic subsamples: 0.89 and 0.90, civil subsamples: 0.89 and 0.92).
Data Analysis
Exhaustive analyses were conducted to evaluate the structure, reliability, and validity of the ISQ for both the civil and forensic inpatient samples. First, an item analysis was conducted in an effort to identify strong and weak items of the ISQ. Second, a principal components analysis (PCA) with varimax rotation was conducted to assess for the presence of underlying factors of the ISQ and a confirmatory factor analysis (CFA) was conducted to confirm the presence of these factors. Third, internal consistency and test-retest reliability were calculated for the PCA and CFA scales. Fourth, the validity of the ISQ was assessed by comparing it with other satisfaction measures and also for PCA and CFA results.
Results
Item Analysis and Screening
Items of the ISQ were analyzed to determine whether two criteria were met: each item of the ISQ had to correlate significantly with a total scale score that comprised all ISQ items minus the respective item and each item of the ISQ had to differentiate between low and high total scores.21 Therefore, for criterion one, several total scale scores were computed, each excluding one distinct item. Then, Pearson's product-moment item-total correlations were computed between each of the total scale scores and the respective item excluded from the total scale score. This procedure was carried out separately on the forensic and civil subsamples. All items for both samples, excluding items 26 (“There should be more opportunities for therapy on the unit.”) and 40 (“I wanted to receive more therapy while I was here.”), correlated significantly with total scale scores. For criterion two, independent-sample t tests determined whether values of each item were significantly different between the lower (sum scores of 196 and below) and the upper (sum scores of 245 and higher) quartile groups. All items differentiated between groups except for items 26 and 40. As these items failed both criteria, they were dropped from further analyses of the forensic and civil samples. This procedure was repeated with the forensic and civil subsamples collapsed into one group.
Principal Components Analysis
The structure of the ISQ with the remaining 39 items was analyzed via the principal component method of factor extraction with varimax rotation (PCA) in a subsample of the forensic and civil samples. The initial iteration of factors for the PCA method sets item communalities (sum of squared loadings for an item across the factors) to 1.00 so that 100 percent of the item's variance is accounted for by the factors. Subsequent iterations use item-squared multiple correlations of each item to compute factor estimates.22 Given the differences in the nature of these inpatient samples, factor analyses were run separately for the forensic and civil inpatient subsamples; however, the method of PCA was identical for both samples. For the initial PCA, the number of components was left unrestricted. Subsequent PCAs restricted the number of factors extracted based on the number of components with eigenvalues of 1.00 or greater, an examination of the Scree plot, and an inspection of item loadings.22 The number of factors extracted was based on the number of items loading 0.40 or higher on that factor, with a cutoff of at least two high loaders. Items were deleted if they did not load 0.40 or higher on any components, or in the event that they loaded 0.40 or higher on more than one component. Changes in the percentage of variance accounted for by the factor structure also informed modifications to the PCA. The results of the final PCAs are depicted in Table 1. PCAs were also repeated on a collapsed group of forensic and civil inpatient subsample group. The resulting factor structure was significantly different from that of the separate subsamples, suggesting that the factor structure of satisfaction is unique for forensic and civil inpatient groups. The factor structures for the forensic and civil subsample groups are described separately in the following sections.
Forensic Inpatient Sample PCA Factor Structure
For the forensic subsample, the final PCA extracted four factors with three or more items loading 0.435 or higher and accounting for 57.75 percent of the variance (Table 1). Mean scale scores were created based on the resulting factor structure (Table 2). Factor 1 (five items) was Medication and Treatment; Factor 2 (four items) was Physical Environment; Factor 3 (three items) was Unit Rules and Procedures; and Factor 4 (four items) was Satisfaction with Activities. The total mean score was also computed and ranged from 1.31 to 4.00, with a mean of 2.59 (SD 0.48).
Although the scales correlated significantly with one another (Table 3), their magnitude was considered weak (0.23) to moderate (0.40). Paired sample t tests were computed to test for differences between the scales (Table 2). Satisfaction was highest for the Physical Environment scale compared with other scales of satisfaction, including Medication and Treatment (t(212) = −4.21, p < .01), Unit Rules and Procedures (t(212) = 4.86, p < .01), and Activities (t(212) = 3.80, p < .01).
Reliability was assessed via the standardized Cronbach's α for the resulting scales. For the forensic subsample, the total scale α was 0.82 and ranged from moderate to high for the individual scales: Medication and Treatment (0.83), Physical Environment (0.70), Unit Rules and Procedures (0.75), and Activities (0.61).
To evaluate concurrent validity of the ISQ, Spearman's correlations were computed between the ISQ scales, PCSQ total score, and total/scale scores of the IESQ (Table 4). All scales correlated positively with one another, with the exception of the Medication and Treatment scale and the IESQ Satisfaction with Environment scale. The ISQ was more strongly related to the PCSQ total score than that of the IESQ. Regarding the scores of the ISQ, the Medication and Treatment and Activities scores were most strongly related to the PCSQ total score. The Unit Rules and Procedures score was highly associated with the staff-patient alliance score of the IESQ.
Civil Inpatient Sample PCA Factor Structure
The final PCA for the civil subsample extracted three factors, with three or more items loading 0.495 or higher and accounting for 49.31 percent of the variance (Table 1). Table 4 displays descriptive information for these scales: Factor 1 (11 items), Needs and Opportunities; Factor 2 (5 items), Food and Comfort; and Factor 3 (2 items), Telephone Access. A total score with the 17 items was computed and ranged from 0.88 to 4.00 with a mean of 2.63 (SD 0.53).
Score intercorrelations were significant and ranged from weak (0.22) to moderate (0.45) (Table 5). Differences in scale scores were evaluated with paired sample t tests (Table 4). The Telephone Access score was significantly higher than those of Needs and Opportunities (t(202) = −2.33, p = .02) and Food and Comfort (t(202) = −2.60, p = .01). The standardized Cronbach's α for the resulting scales were high: Needs and Opportunities = 0.84, Food and Comfort = 0.71, and Telephone Access = 0.77. Reliability for the total scale score was 0.86.
As with the forensic sample, Spearman's correlations among the ISQ, PCSQ, and IESQ scales were computed to evaluate concurrent validity of the ISQ (Table 5). The ISQ was strongly related to both the PCSQ and IESQ. Most correlations were moderate with the exception of the correlations that involved the ISQ Telephone Access scale. These correlations, although significant, are considered weaker (ranging from 0.22 to 0.35) in comparison with other ISQ scale correlations.
Confirmatory Factor Analysis
The factor structures obtained by PCA of the forensic and civil sample data were tested via confirmatory factor analyses (CFA) with the generalized least-squares (GLS) model estimation technique, with calculations performed by AMOS 16.0.23 Several indices were used to assess the fit of the models: the conventional chi-square statistic, with nonsignificant p values indicative of good fit; the chi-square/degrees of freedom (df) ratio with values under 2.00 representing adequate fit24; goodness-of-fit (GFI),25 with values closest to 1.00 considered adequate; root mean square error of approximation (RMSEA26,27), with values of 0.05 and under indicative of good fit; and Baye's information criterion (BIC), used for model comparison, with lower values representing a better fit.28 Modification indices, parameter estimates, squared multiple correlations (SMCs), and the standardized residuals matrix were evaluated to inform decisions regarding the changes needed to improve the model. Guided by the modification indices, items were either dropped or moved when parameter estimates were nonsignificant, SMCs were low, and/or standardized residuals were 2.58 and higher.29 In addition, before the a priori model was revised, items and their respective factors were inspected to ensure that suggested changes were theoretically sound. The forensic and civil a priori factor structures were tested separately.
Forensic Inpatient Sample CFA Factor Structure
For the forensic subsample (F-ISQ), the four-factor, 16-item structure was tested (Table 1). The model reached minimization and was recursive. However, the fit of this model was weak, with several nonsignificant parameters, large residuals, and poor fit indices. Therefore, revisions were made to the a priori model in an effort to improve the model's fit and uncover a more streamlined model of satisfaction in the forensic subsample. Several items were dropped because of nonsignificant parameter estimates and/or low SMC values, including items 1 and 3 from factor 4. Modification indices indicated that item 25 loaded better on factor 4 than on factor 1. Since the remaining items of factor 1 focused on medication-related aspects of hospitalization, item 25 was moved to load on factor 4. After making these changes, we conducted a separate CFA. This revised model was recursive and reached minimization after eight iterations, displaying improved and adequate fit with the data. The fit of the model was significantly improved, with a nonsignificant chi-square value (χ2(84) = 105.76, p = .05), lower chi-square/degree of freedom ratio (χ2/df = 1.26), higher GFI (GFI = 0.932), an RMSEA value close to 0 (RMSEA = 0.035, 90% confidence interval = 0.000–0.055), and a significantly lower BIC than the CFA of the a priori model (BIC = 395.40).
This final model, displayed in Table 1, included four factors and 14 items (factor 1, four items; factor 2 indicated by four items, factor 3 indicated by two items, and factor 4 indicated by four items). The factors were Medication, Physical Environment, Telephone Access, and Unit Rules and Procedures. Satisfaction was highest on the Physical Environment scale compared with Medication and Treatment (t(207) = −3.85, p < .01), Telephone Access (t(207) = 2.79, p = .01), and Unit Rules and Procedures (t(207) = 7.85, p < .01). The reliability of these scales was considered moderate, with a standardized Cronbach's α of 0.81 for the total score and scaled α of 0.79 for Medication, 0.77 for Physical Environment, 0.75 for Telephone Access, and 0.70 for Unit Rules and Procedures. To demonstrate concurrent validity, CFA factors were correlated with the total and scale scores of the PCSQ and IESQ (Table 3). The significant associations between the total score and the total PCSQ and IESQ scores were considered weak to moderate; slightly stronger validity was demonstrated with the revised-factor structure.
Civil Inpatient Sample CFA Factor Structure
The a priori factor structure uncovered via PCA for the civil subsample (C-ISQ) was also tested by CFA in a separate civil subsample. This model reached minimization and was recursive. The model demonstrated several large residuals and low SMCs, resulting in relatively poor fit indices. Therefore, the model was revised by dropping items with residuals above 2.58 (items 16, 17, 19, and 20), and items with low SMC values (items 1, 21, and 22) (Table 1). As such, two factors remained and were fit to the data via CFA. This revised model was recursive and reached minimization after 19 iterations. The fit of the model was significantly improved across all indices (χ2(53) = 64.56, p = .13; χ2/df = 1.22; GFI = 0.947; RMSEA = 0.033; 90% confidence interval =.000-.058), and BIC was 259.52. This final adapted model included two factors and 11 items (Table 1). Factor 1 (seven items), was Therapeutic Needs and Factor 2 (four items) was Attention and Comfort. A paired-sample t test revealed similar levels of satisfaction between the Therapeutic Needs scale and the Attention and Comfort scale (t(207) = 0.12, p = .91). The reliability of these scales was considered moderate to high, with Cronbach's α of 0.85 for the total scale, 0.70 for the Attention and Comfort scale, and 0.80 for the Therapeutic Needs scale. To demonstrate concurrent validity, CFA factors correlated moderately with the total and scale scores of the PCSQ and IESQ (Table 5). The total score correlated significantly with the total PCSQ and IESQ scores and was considered moderate in magnitude.
Multiple Group Comparisons
We were also interested in determining whether the final factor structures confirmed via CFA differed between the civil (C-ISQ) and forensic (F-ISQ) inpatient samples. Therefore, multiple group comparisons were computed by using AMOS 16.0. This method evaluates whether the factor structure for one sample also fits the data from another sample. (For example, does the forensic sample factor structure of inpatient satisfaction also work with the civil sample data?) When the civil factor structure was fitted to the forensic sample, the equal (constrains pathways of structure to load equally across groups) and unrestricted (allows pathways to load freely for both groups) versions of the model were not significantly different from one another (χ2diff(11) = 9.78, p = .55). However, when the forensic factor structure of patient satisfaction was fitted to the civil sample, the unrestricted version of the model was significantly different from the equal loadings version (χ2diff(16) = 27.18, p = .04). Inspection of the fit indices supported the unrestricted version of the model over the equal loadings version. Therefore, the forensic inpatient factor structure of the ISQ (F-ISQ) does not function in a civil inpatient sample and more sufficiently represents the factor structure of the ISQ in a forensic sample. In other words, while the civil inpatient factor structure of the ISQ is equally adequate in both civil and forensic inpatient samples, the forensic factor structure of the ISQ is only adequate in the forensic sample.
Discussion
The goal of the present study was to develop and investigate the psychometric properties of a survey designed to evaluate patient satisfaction on forensic and civil psychiatric inpatient units. A new 41-item questionnaire was subjected to an item analysis, principal components analysis, and confirmatory factor analysis in an effort to develop a psychometrically sound measure of patient satisfaction in civil and forensic inpatient psychiatric populations. Results support two forms of the Inpatient Satisfaction Questionnaire (ISQ) that assess patient satisfaction. (See Table 1 for items and respective scales.) For a civil inpatient psychiatric population, an 11-item questionnaire (C-ISQ) addressing two domains of satisfaction, Therapeutic Needs and Attention and Comfort, was considered a reliable and valid measure. For a forensic inpatient psychiatric population, a 14-item questionnaire (F-ISQ) addressing four domains of satisfaction—Medication and Treatment, Physical Environment, Telephone Access, and Unit Rules and Procedures—was considered a reliable and valid measure. Notably, the 11-item C-ISQ can also be used in a forensic population; however, the 14-item F-ISQ does not apply to civil populations. Therefore, the value of one version of this measure of satisfaction (i.e., F-ISQ) is its utility for exclusive use with forensic inpatients, although another version (i.e., C-ISQ) can be used in civil and forensic patient samples and also allows satisfaction to be compared across these types of psychiatric units.
In terms of the domains of satisfaction assessed on the ISQ, the final CFA structures resulted in five different scales across the two samples. These scales tapped different areas of the inpatient's experience. For forensic inpatients, satisfaction was divided into four primary factors: Medication (understanding why medication is being prescribed, receiving medications in a timely manner, having medication side effects adequately explained, and comprehending the importance of taking medication), Physical Environment (bathroom, public and patient rooms, showers), Telephone Access (access to and time spent using the phone), and Unit Rules and Procedures (seeing staff in a timely manner, discussing the treatment plan, understanding unit rules, and having rights explained). Of note, the third factor emerging from the CFA was a revision of factor 4 from the PCA, Satisfaction with Activities. Telephone Access was separated from an item assessing satisfaction with getting enough to eat and having opportunities to exercise. The result underscores the value of telephone access for this type of patient population. Perhaps the importance of telephone access emerged because of the phone restrictions that are enforced in correctional institutions.
For civil inpatients, satisfaction was divided into two primary factors: Therapeutic Needs (having medication side effects and patient rights adequately explained, believing that there are enough therapy opportunities on the unit, receiving sufficient psychotherapeutic treatment, perceiving the unit as a good place to discuss problems, and believing that medications are used to help calm patients) and Attention and Comfort (perceiving the staff as helpful, sufficient opportunities to engage in activities, feeling comfortable on the unit, and getting enough to eat). Comparisons between items and respective scales on the C-ISQ and F-ISQ reveal that only one (item 29, having rights explained to them) was actually relevant in both samples, loading on the F-ISQ's Unit Rules and Procedures and C-ISQ's Needs and Opportunities scales. In fact, the construct of satisfaction among civil and forensic inpatients as measured by the ISQ is characterized by distinct factors. Results suggest that ISQ satisfaction is composed of different domains for civil and forensic psychiatric inpatients.
Some of the ISQ scales are similar to those from other inpatient satisfaction measures; however, the differences are notable. For example, while other measures assess treatment satisfaction, many do not focus specifically on patients' satisfaction with medication.3,16,18 Such satisfaction is arguably a central part of treatment on an inpatient psychiatric unit and is highly relevant to the question of maintaining treatment compliance and success following discharge from the hospital. Forensic patients especially may find that greater attention is paid to the basic process of medication administration in the inpatient setting than in the correctional setting (e.g., whether hospital nursing staff approach patient rooms or patients are called individually to receive their medications). Thus, it can be hypothesized that forensic patients who feel satisfied with medication procedures in the hospital may be more likely to comply with treatment if and when they return to their correctional setting, where medication administration may be more lax. Further, patients who are satisfied with medication procedures in the hospital may also be more likely to maintain compliance on their return to the community, which could also work to reduce the likelihood of recidivism and/or hospital readmission in some cases. Collectively, it is suggested that working to increase patient satisfaction as it relates specifically to medication may lead to increases in future treatment compliance.
Other scales assess satisfaction with the patient's environment. For example, the IESQ has a specific scale assessing Satisfaction with Environment. Although the Physical Environment scale score of the F-ISQ did not strongly correlate with this scale score, the IESQ scale also includes assessment of items beyond principal environmental concerns (e.g., activities provided, occupational therapy groups). It is important that unit staff consider aspects of the physical environment that may be dissatisfying to patients, as these factors can contribute to patient aggression.30 For the present sample, this was the highest domain of satisfaction of the forensic inpatients. There are several interpretations one can consider regarding this finding (e.g., simple differences in esthetics, more exposure to treatment staff than to correctional officers, lower noise levels, and greater freedom of movement compared with correctional institutions). In addition, jail and prison overcrowding may account for the highest satisfaction rating on the Physical Environment scale in an inpatient unit, as some patients on the forensic unit are able to have their own rooms, and one room is shared by three patients, at most.
Study Limitations
Despite the benefits of developing a brief and psychometrically sound measure to assess patient satisfaction with both civil and forensic inpatient populations, there are limitations. As previously mentioned, there are differing opinions on the value of obtaining patient opinions, and one must consider the possibility that responses to items on the ISQ may not reflect the true opinion of the patient. This possibility is especially important to consider with the F-ISQ, as forensic patients may have a strong desire to remain in the hospital as opposed to being discharged and returned to their respective correctional institutions. Moreover, the sample used in the present study was nonrandom, since data were collected during certain specific times of the year. It could be argued that the patient population differs over time. In addition, since the goal of the ISQ is to obtain the patients' degree of satisfaction with their care, it would be useful to consider at what point in the duration of their care they are asked to complete the ISQ. Oftentimes patients who are about to be discharged are asked to complete satisfaction surveys; however, choosing a fixed time to measure patients' attitudes may indeed affect ratings (e.g., at discharge, patients' conditions have improved, they are leaving the hospital, and may give more positive reviews). It would be valuable if future research had a truly random administration of surveys reflecting different times of the year and different points during the patients' stays.
Another possible limitation worthy of mention concerns the C-ISQ, which assesses patient satisfaction across only two domains (Needs and Opportunities, Food and Comfort), while the F-ISQ assesses patient satisfaction across four domains. The F-ISQ cannot be validly applied to civil inpatient populations and this restriction invariably limits the domains of patient satisfaction that can be reliably assessed on civil units when the C-ISQ is used in isolation. Because the C-ISQ can be used in forensic inpatient psychiatric settings, it allows for broader and more comprehensive assessment of patient satisfaction among forensic inpatients. As the authors do not have knowledge of any existing, psychometrically sound instruments that are designed to evaluate patient satisfaction among forensic inpatients, the fact that both versions of the ISQ (forensic and civil) can be applied to forensic inpatients is considered a significant contribution to the proper assessment of patient satisfaction in this setting.
Conclusions
Collectively, research has demonstrated the value of assessing patient satisfaction. However, the development of psychometrically sound measures to accomplish this evaluation has been minimal, and to our knowledge, absent in the domain of forensic inpatient psychiatry. As greater attention is paid to the importance of ensuring the delivery of high-quality mental health services to both civil and forensic populations, a valid and reliable measure that can be used to assess patient satisfaction with both populations is needed. Findings demonstrate the utility of the F-ISQ and C-ISQ as valid, reliable, and brief instruments that can serve as rapid assessments of patient satisfaction (the C-ISQ with both civil and forensic inpatient psychiatric populations). Findings of such instruments can be used to increase the quality of service delivery, treatment compliance, and future use of mental health services. Utilizing patient satisfaction ratings for staff training and treatment program development may increase treatment compliance and patient functioning and increase the likelihood that patients will seek out mental health services following discharge from the hospital or release from jail or prison.
Acknowledgments
The authors thank J. Reid Meloy, PhD, for reviewing earlier drafts of the article and the numerous graduate assistants who helped in the data collection process.
- American Academy of Psychiatry and the Law