Abstract
Background
To date, no pilot questionnaire of perioperative anesthetic care has been developed based on psychometric methodology in Taiwan. We describe the development and qualitative and quantitative validation of a pilot version of a psychometric questionnaire designed to measure patient satisfaction with perioperative anesthetic care in Taiwan.
Methods
A rigorous protocol was followed and involved expert consultation, literature review, development of the interview guide, semi-structured in-depth interviews, pretest and application of Aiken's two quantitative methods to determine the content validity coefficient (V value) and homogeneity reliability coefficient (H value) of each item, and the questionnaire as a whole, to ensure the pilot questionnaire showed high-content validity and reliable homogeneity.
Results
Our final pilot questionnaire contained six dimensions with 32 items; five of the domains were similar to those reported in the literature. One dimension entitled anesthesia-related sequelae was new. The V values for the 32 items ranged from 0.80 to 0.97 (p < 0.01) and the mean value (V —) of the questionnaire as a whole was 0.90 (p < 0.01). The H values ranged from 0.55 to 0.88 (p < 0.01) and the mean H value (H—) of the questionnaire as a whole was 0.71 (p < 0.01).
Conclusion
We have developed a valid and reliable pilot questionnaire to measure patient satisfaction with perioperative anesthetic care in Taiwan. The final version of the pilot questionnaire is a 32-item instrument with six dimensions, one of which, anesthesia-related sequelae, is new.
Keywords
anesthesia; patient satisfaction; pilot projects; questionnaires; Taiwan;
1. Introduction
Patient satisfaction has been highlighted as an important objective of healthcare, a marker for outcome quality and a useful indicator for service quality.1−3 The concept of satisfaction is very com-plicated and is not clearly comprehensible. It is mul-tidimensional in nature and is influenced by physical, cognitive, affective and sociodemographic factors, as well as intrinsic psychological traits and cultural attitudes.4,5 Moreover, established satisfaction ques-tionnaires need to be revised and updated regu-larly to ensure high content validity.6
A rigorous protocol should be used to develop effective questionnaires. The first stage is the de-velopment of a pilot questionnaire based on the results of comprehensive literature reviews, expert consultation, face-to-face in-depth interviews, and pretest and analysis of the content validity and re-liability of the pilot questionnaire. The second stage involves two commonly used statistical techniques, exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), to validate the pilot question-naire developed in the first stage. However, content validity of a pilot questionnaire must be established before conducting EFA and CFA. Hair et al7 em-phasized that content validity is perhaps the most important validity test in the processes of ques-tionnaire development.
Here, we developed a highly standardized, patient self-reported and multidimensional psychometric pilot questionnaire to assess patient satisfaction with perioperative anesthetic care in Taiwan. The pretest was evaluated qualitatively by measuring its face validity. Content validity and homogeneous reliability were evaluated quantitatively by ex-amining the strength of Aiken’s content-validity coefficient (V value) and homogeneity reliability coefficient (H value).8
2. Methods
Patients aged over 18 years who had undergone elective surgery, of any form, and who were knowl-edgeable and could communicate in either Mandarin Chinese or Taiwanese were eligible for this study. We explained the purposes, methods, and expected study benefits. All patients provided written in-formed consent before enrolment in the study.
The protocol was approved by the Institutional Review Board of our hospital before starting the study. The protocol is outlined in Figure 1. We stressed to the patients that, at present, there is no reliable or practical satisfaction scale of an-esthesia care to evaluate the quality of anesthesia service in Taiwan. To satisfy the strong desire of patients and their families to obtain a high-quality anesthesia service, we need the help of the patients to develop a pilot questionnaire of patient satisfac-tion with anesthesia care through this study, which is specific for Taiwanese culture and provides ad-equate reliability and validity.
We incorporated the input from six anesthesiol-ogists, four nurse anesthetists, two sociologists and one statistician to identify the possible items to be included in the interview guide based on their lit-erature reviews. The same professionals were also involved in ongoing revisions of the pilot question-naire. We searched the databases EMBASE, PubMed, MEDLINE-Ovid, The Cochrane Library, CINAHL and PsycINFO for references. Only four satisfaction ques-tionnaires related to perioperative anesthetic care have been developed using psychometric tech-niques.4,5,9,10 We reviewed these four reports and other studies related to survey and summarization of the findings of perioperative patient satisfac-tion.1−3,11−19 We were of the opinion that the fol-lowing five dimensions should be included in the interview guide: information (including patient in-volvement in decision-making), discomfort and needs (including professional competence), provider−patient relationships (including attention, privacy, confidence and respect), waiting period (including delays and services) and fear and concern (includ-ing pain management and anxiety).
In-depth interviews with patients (Group 1) were conducted by a trained interviewer. Patients were interviewed individually by the interviewer and a nurse in the preanesthesia consultation clinic (PAC). The interview was divided into two parts. In the first part, patients were asked to freely de-scribe their satisfaction with anesthetic care to identify possible dimensions and items to be in-cluded in the pilot questionnaire. The second part of the interview was based on the interview guide containing items from the five possible dimensions that emerged from the literature reviews and ex-pert opinions, as described above.
The interviewer emphasized the impact of peri-operative performance of anesthetic care on patient satisfaction by referring to the theory of expec-tation.20,21 In 1988, Parasuraman et al22 proposed the theory of expectation and defined satisfaction as the discrepancy between expectation and cur-rent life experience. Therefore, the views of the patients were gathered regarding the degree of discrepancy or the degree of congruence between their expectations of and their perceived expe-rience with anesthetic care in the perioperative period.23,24
At the end of the interviews, patients were asked an open-ended question designed to allow them to give their opinions about any missing 182 W.C. Mui et alitem(s). Interviews were continued until no new ideas emerged and no further modifications of items were needed in up to a total of 20 patients.5 The first version of the pilot questionnaire was developed mainly according to the results of the semi-structured interviews.
We pretested the face validity, comprehensibil-ity and readability of the first version of the pilot questionnaire in a new sample of patients (Group 2) in the ward within 48 hours after their elective surgery under anesthetic care (general anesthesia, regional anesthesia or monitored anesthesia care).
The second version of the pilot questionnaire was developed after the pretest to allow for incorpora-tion of the modifications indicated for the first ver-sion of the pilot questionnaire.
To collect quantitative evidence to evaluate the content validity and homogeneity reliability of our pilot questionnaire, we used Aiken’s two quantita-tive methods to calculate V and H values for each item in the second version of the pilot question-naire and for the questionnaire as a whole.25,26 Data were collected in the ward from a new group of patients (Group 3) within 48 hours after their elective surgery under anesthetic care using a five-point Likert scale to measure the importance of each item. We included items with significantly strong (p < 0.01) V and H coefficient values from the transformed second version of the instrument in the final version of the pilot questionnaire.
The techniques used to calculate and determine the statistical significance of the V and H coefficient values are described in Appendix 1.
Download full-size image
3. Results
This study was conducted between July 2008 and January 2009. A total of 95 patients met the inclu-sion criteria and participated in the semi-structured in-depth interviews. The sociodemographic data, history of anesthetic care and ASA physical status of the patients who participated in the interviews (Group 1), the pretest (Group 2) and the Aiken’s V and H values testing (Group 3) are shown in Table 1.
The results of the semi-structured interviews revealed that the five dimensions in the interview guide were compatible with the expectations re-ported by the patients.
A new dimension was developed based on the results of the interviews with 95 patients who had received anesthesia care within the last 3 months. We termed this dimension as “anesthesia-related sequelae”. This new dimension was initially cate-gorized under the dimension of “fear and concern”. However, after completing the in-depth interviews with the 95 patients, we found that the concept of this title was sufficiently distinguishable to be in-dependent and was thus designated as a separate dimension. We found that many patients had er-roneous expectations and groundless fear of the sequelae caused by anesthetics or anesthesia pro-cedures. For example, 52 of our 95 patients in the interviews believed that intraspinal (epidural or spinal) administration of anesthetic agents to in-duce regional anesthesia prior to surgical interven-tion or for postoperative pain management would induce sequelae that could impair their working capacity and hamper their daily activities after surgery, such as short- to long-term, moderate to severe lower back pain, or even hemiplegia. In fact, 25 patients attributed the cause of their chronic lower back pain to insertion of the spinal needle into their “dragon bone” () (Taiwanese dialect for vertebral column) for administration of anesthetic agents during a previous surgery. These 25 patients (15 female, 10 male and 13 aged > 65 years) requested the anesthetist (the interviewer) in the PAC that no dragon bone injection should be considered again for their forthcoming surgery.
Moreover, during the interviews, 44 of the 95 patients believed that the administration of mor-phine or other anesthetic agents to induce general anesthesia or for tracheal intubation or mechani-cal ventilation during surgery could lead to serious complications and sequelae such as brain injury, mental retardation, drug addiction, liver damage or other vital-organ injury. The patients considered that anesthesia-related sequelae would occur for a certain period of time after their operation and might impair their daily activities due to the ad-verse effects of anesthetics on their mentation and physical capacity. Therefore, we added four more items to this dimension in the pilot version of the questionnaire. In total, 38 items were collected for the first version pilot questionnaire. We have only presented the results of the final version of the pilot questionnaire to avoid confusion with the earlier versions of the questionnaire.
The 38 items in the first version of the pilot ques-tionnaire were pretested in a new sample of 34 patients (Group 2) in the ward within 48 hours after their elective surgery under anesthetic care. These 34 patients were asked to evaluate the 38 items for face validity, comprehensibility and readability. Three items were deleted after the pretest because of re-peatability and insignificance. Therefore, the second version of the questionnaire consisted of 35 items.
Aiken’s content-validity coefficients (V) and homogeneity reliability coefficients (H) were then calculated for each of the 35 items in a new group of 30 patients (Group 3) who were selected within 48 hours after their elective surgeries under anes-thetic care. These 30 patients were asked, while in the ward, to judge the importance of each item on a five-point scale, ranging from 1, extremely unim-portant, to 5, extremely important.
The V coefficients for each item ranged from 0.39 to 0.97 (Table 2), the V— value was 0.86 and the Z-score of the V— value of the questionnaire as a whole was 34.38 (p < 0.01). The H coefficients for each item ranged from 0.46 to 0.88, the H— value was 0.69 and the Z-score of the H— value of the questionnaire as a whole was 25.14 (p < 0.01). The high V— and H— values, and the statistically significant Z-scores, indicated that the second version of the pilot questionnaire, when analyzed as whole, is a valid and reliable in-strument for measuring patient satisfaction with anesthetic care. Nevertheless, when each item was analyzed individually, the V values for items 8, 22, and 31 had right-tail distribution probabilities of less than the 0.01 level of significance (V < 0.62). There-fore, we deleted these three items, resulting in a 32-item pilot questionnaire. The V— and H— values for the final version were 0.90 and 0.71, respectively. Both values were statistically significant (Table 2).
In summary, we have developed a valid and re-liable pilot questionnaire to measure patient satis-faction with perioperative anesthetic care in Taiwan. The 32-item instrument contains the following six dimensions: information (items 1−5), discomfort and needs (items 6−9), provider−patient relation-ship (items 10−17), waiting period (items 18−21), fear and concern (items 22−28), and anesthesia-related sequelae (items 29−32) (Appendix 2).
4. Discussion
This is the first pilot questionnaire of patient satis-faction with perioperative anesthetic care designed using formal psychometric methodology for use by patients in Taiwan. This is also the first study to measure content validity and reliability of each item and the questionnaire as a whole using Aiken’s content-validity coefficients (V) and homogeneity reliability coefficients (H) to develop a pilot instru-ment to measure patient satisfaction with periop-erative anesthesia care.
Content validity is the assessment of the degree of correspondence between the items selected to constitute a measurement instrument and its con-ceptual definition.7 Content validity is assessed through ratings by experts, pretest with multiple subpopulations, or other means. Assessment of the content validity of the pilot questionnaire is the most important validity test in the process of ques-tionnaire development.7 Content validity must be established before questionnaire validation can be performed.
Decisions regarding the content validity of sin-gle items or the complete questionnaire are tra-ditionally described in terms of dichotomies (yes/no, valid/invalid) or ranks (high validity/moderate validity/low validity). These decisions rely on sub-jective logical analysis and rational assessment by the experts. Researchers often pay little atten-tion to the statistical nature of these decisions. Furthermore, the question about how reliable such decisions are may be entirely neglected.25,26 There-fore, it seems necessary and helpful to apply more objective and appropriate statistical methods to analyze data from content validity judgments on rating scales and other ordinal psychometric in-struments. Aiken offered a set of procedures for computing and determining the statistical signi-ficance of a content validity coefficient (V value) and a homogeneity coefficient (H value) for such purposes. These procedures, which use the multi-nomial probability distribution for small samples and normal curve probability estimates for large samples (n > 25), can be used in a variety of situa-tions where judgments of the content validity of items or questionnaires are made on ordinal rating scales.25,26
When the V and H values of each single item and of the entire questionnaire reach the level of significance, we have more objective and quanti-tative evidence to state whether the items and the questionnaire show high content validity and in-ternal consistency. In this study, we deleted three items (items 8, 22 and 31) from the second version of the pilot questionnaire because the V values of the three items were not significant (see Table 2). This revision resulted in a more objective final ver-sion of the questionnaire (Appendix 2). Our study is the first report to describe the development of a pilot questionnaire to assess patient satisfaction with anesthetic care based on objective statistical evidences.
Five dimensions in this questionnaire were adopted from literature reviews and expert opin-ion and were consistent with the expectations re-ported by the patients in semi-structured interviews. In addition, a new dimension was revealed by these interviews. Many of the patients in our study ex-pressed concern about anesthesia-related seque-lae after their surgeries. Furthermore, about half of the patients reported having an irrational fear that morphine administration for general or regional anesthesia, or for postoperative pain management, would induce severe toxic effects or addiction. This irrational belief and fear has its beginnings in the Opium War (), a humiliating war in Chinese history. Moreover, morphine is a control-led drug in Taiwan, and the mandarin translation of “controlled drug” is similar in meaning to “toxic drug” in English. In addition, more than half of the patients in our study believed that the penetration of the dragon bone by an intraspinal needle would lead to postoperative chronic back pain or other sequelae. The dragon bone is considered to be the most precious bone in the body in Taiwanese cul-ture. These historical and sociocultural factors might contribute to the irrational fear of anesthetic care in Taiwanese patients.
The acceptance of prejudiced information from the patients’ relatives or friends who had experi-enced complications or sequelae from anesthesia may provide another reason for the misconception of anesthetic care in these Taiwanese patients. This may be partly due to long-term staffing shortage of anesthesia specialists in Taiwan or to the unsound national health care system and regulations. The current Physician Practice Act of Taiwan stipulates that a licensed physician is allowed to perform an-esthesia without offense against any Medical Care Law. It was not until July 1999 that the Department of Health, Taiwan announced that only anesthesia specialists can apply for reimbursement of general anesthesia from the National Health Insurance Scheme. However, at the end of 2008, there were only 916 serviceable anesthesia specialists in Taiwan, despite Taiwan’s population being approximately 23 million and nearly 0.9 million procedures re-quiring anesthesia each year in Taiwan (informa-tion from the Taiwan Society of Anesthesiologists through oral and written communication). Accord-ingly, many anesthesia services are still performed by non-anesthesia specialists. If we want to elimi-nate this well-established fear among Taiwanese pa-tients for anesthetic care, we believe that we need to revise the medical laws and regulations (e.g., all anesthetic procedures should be legally per-formed by anesthesia specialists only, irrespective of National Health Insurance reimbursement) and increase anesthesia staffing resources. Similarly, we should further improve our techniques in per-forming epidural and spinal anesthesia and improve the quality of general anesthesia to reduce the fear of anesthesia. As a result, the concerns reported by Taiwanese patients regarding anesthesia-related sequelae could be reduced.
In the development of this pilot questionnaire, it was our intent to let the three groups of patients (i.e., monitored anesthesia care [MAC], general anesthesia [GA] and regional anesthesia [RA]) be subjected to interview simultaneously, to develop a broader measurement scale. We defined MAC, GA and RA as the period starting with the attend-ance of the patient at the PAC, the operation it-self, transfer through the postanesthesia room, and ending 2−3 days after being returned to the ordi-nary ward. Whether this definition is correct, or whether these three groups can be represented by a single domain or can be represented by more than one domain, our future research will entail the use of statistical methods to analyze and extract the underlying factor structure by EFA and the meas-urement model and structural model by CFA to measure the construct validity, convergent relia-bility and measurement invariance to validate the questionnaire. To our knowledge, the use of EFA and CFA to develop a satisfaction scale has not yet been attempted in the anesthesia arena.