Development and validation of the Web-Based Patient-Reported Outcomes Capture System-Needs, Acceptance and Readiness Assessment (WPROCS-NARA) questionnaires

Introduction

Symptom monitoring and management and long-term follow-up of late toxicity are essential in improving outcomes among cancer patients and survivors. Prompt communication and intervention for distressing symptoms among elderly and terminally-ill patients are often impeded by limited ambulatory capacity and access to healthcare facility. Transportation costs and logistics commonly deter long-term follow-up for cancer surveillance and late toxicity monitoring and management, particularly for patients from remote areas. At our institution, up to 50% of cancer patients come from remote provinces, 40% are aged 60 years and above, and 30% have advanced-stage cancers.

Information and communications technology (ICT) applications have been employed to overcome above difficulties. For such applications to be truly useful however, user-centered design approaches and appreciation of the socio-technical aspects of complex systems are both important (1). Furthermore, it is necessary to assess the needs and capacity of the intended users (Framework of Dissemination in Health Services Intervention Research) (2). At the individual level, there are two dimensions of capacity or readiness for change—psychological (appropriateness, change efficacy and valence) and structural (knowledge, skill and ability alignment) (3). Data on factors related to health, health status and healthcare access that would influence potential usage of health-related information and communication technology (needs); perception and attitude of cancer patients towards use of such a technology (psychological readiness, hereafter referred to as acceptance); and access to and usage of the computers, the Internet and mobile telecommunication devices (structural readiness, hereafter referred to as readiness).

The investigators sought to develop and validate an instrument to survey the needs, acceptance and readiness for a web-based monitoring system for patient-reported outcomes (PROs), among adult cancer patients.

Methods

The study was approved by the University of Santo Tomas Hospital Institutional Review Board (USTH-IRB). All expert reviewers and participants provided written consents prior to inclusion in the study. The methodology is summarized in Figure 1.

Figure 1 Methodology.

Literature search and modification of instruments

We searched the literature for valid instruments that could survey the needs, acceptance and readiness for a web-based PROs monitoring system among adult cancer patients. We used the keywords “information and communication technology” AND “patient-reported outcomes” AND “cancer” AND “healthcare needs” OR “acceptance” OR “perception” OR “attitude” OR “readiness” OR “access” OR “usage” to search the following databases: Medline, PubMed, Google and Google Scholar.

The search yielded three applicable instruments, one for each dimension (needs, acceptance and readiness). None of these instruments were specifically developed for cancer patients. A focused group discussion was conducted among the investigators (WR Bacorro and TT Sy Ortin, oncologists; RM Gutierrez, oncology nurse; and JC Que, pain and palliative care specialist) in order to modify the instruments according to our objectives.

Face and content validation and development of a Filipino version

The modified questionnaires were reviewed by a panel of experts that consisted of two oncologists, a pain and palliative care specialist, and an oncology nurse, each with at least 5 years of oncology or oncology-related practice. The modified Delphi technique was employed (4). Each expert independently evaluated the questionnaires, rated the relevance of each item using a 5-point Likert scale, and provided comments and recommendations for revisions pertaining to question content or format. Items with mean score of <3.5 and a median score of <4 were revised accordingly (5). Consensus was reached after a round of critique and revision.

The questionnaires were administered to ten adult cancer patients who were fluent in English. Each participant rated how easy was each item to understand and to answer, using a 4-point Likert scale, and provided comments and recommendations for revisions. Filipino versions were then developed using standard translation procedures and similarly reviewed for face and content validity by ten adult cancer patients fluent in Filipino.

Pilot testing for construct validity and internal consistency reliability

The questionnaires were then administered, in either Filipino or English, to 130 cancer patients for construct validity and internal consistency reliability testing. Recruited participants were at least 18 years of age seen at the USTH-BCI from January 1, 2014 to December 31, 2014.

Statistical analyses

Data gathered were encoded and analyzed using Excel 2013 and Stata 11. Results of face and content validation were analyzed and summarized using descriptive statistics. Responses to the pilot testing were analyzed using principal component analysis. Only components that account for at least 70% total cumulative variance, with eigenvalues >1, and significant scree test results were considered meaningful components or scales. Complex items which loaded or achieved 0.40 loading factor for more than 1 meaningful component were not included in the final questionnaires (6). Revised questionnaires’ internal consistency reliability were estimated using Cronbach coefficient alpha. An alpha of ≥0.7 was considered as acceptable reliability (7).

Results

Development of the survey instruments

Three separate instruments were found: Consumer Assessment of Healthcare Providers and Systems, for needs assessment (8); unnamed questionnaire by Morris and Venkatesh (9), for acceptance assessment; and 2011 Eurostat model questionnaire for community survey on ICT usage in households and by individuals (version 3.1) (10), for readiness assessment. Review of the questionnaires resulted in modification of content (subtraction) and question or response format (local adaptation, such as for currency) according to our objectives and relevant literature. The three resulting questionnaires were named Needs Assessment Questionnaire (NAQ), Acceptance Assessment Questionnaire (AAQ), and Readiness Assessment Questionnaire (RAQ).

Face and content validation, adaptation, and development of a Filipino version

Expert validation revealed that all items were relevant, except for an item in the AAQ, which was considered irrelevant by two evaluators. The latter was omitted. No revisions pertaining to question format were received.

For the patient validation of the instrument, one item in the AAQ and two items in the RAQ in the English version required revision. In the Filipino version, two items in the NAQ required revision. For these items, minor revisions were made in phrasing.

The results of the expert and patient validation are summarized in Figure 2.

Figure 2 Results of face and content validation of WPROCS-NARA questionnaires. (A) Expert validation; (B) patient validation of the English version; (C) patient validation of the Filipino version. WPROCS-NARA, Web-Based Patient-Reported Outcomes Capture System-Needs, Acceptance and Readiness Assessment; NAQ, Needs Assessment Questionnaire; RAQ, Readiness Assessment Questionnaire; AAQ, Acceptance Assessment Questionnaire.

Pilot testing for construct validity and internal consistency reliability

A total of 130 respondents took the survey. Median age was 44 (range, 28–80). The majority were aged less than 60 years old (90%), females (77%), and with secondary schooling (98%).

A total of 14 (30.43%) components were found in the 3 questionnaires—7 in NAQ, 3 in RAQ and 4 in AAQ, as shown on the Scree plot (Figure 3). A summary of these components is found in Table 1 below.

Figure 3 Scree plots of the eigenvalues of WPROCS-NARA questionnaires. WPROCS-NARA, Web-Based Patient-Reported Outcomes Capture System-Needs, Acceptance and Readiness Assessment; NAQ, Needs Assessment Questionnaire; RAQ, Readiness Assessment Questionnaire; AAQ, Acceptance Assessment Questionnaire.

No complex item was determined and eliminated upon principal component analyses. Also, all questionnaires were found to have acceptable reliability (NAQ Cronbach alpha =0.84; RAQ Cronbach alpha =0.77; AAQ Cronbach alpha =0.71).

The final survey instruments

The NAQ consisted of five sections exploring the following aspects in the last three months: (I) overall and emotional health, need for home care, urgent care, and hospitalization, and need for home care (9 items); (II) symptoms and distressing symptoms, and pain and symptom medications (7 items); (III) weight change, problems with eating and need for tube feeding (3 items); (IV) quality of primary doctor and social support, source of professional health care and advice outside the hospital, and ease of access to health care or advice (6 items); and (V) ambulatory capacity, disability, and ease of access to the closest clinic or hospital, our hospital (USTH-BCI) and the primary doctor (5 items). Questions were constructed in single- or multiple-response multiple-choice formats.

The AAQ consisted of two sections exploring the following aspects: (I) age, gender, address, annual individual income and highest educational attainment (5 items), and (II) attitude towards WPROCS, perceived logistic capacity and competency to use such a system, and willingness to acquire materials, equipment, knowledge and/or access to the system when lacking (7 items). For the first section, questions were constructed in single-response multiple-choice format. For the second section, questions employed 4-point Likert scales.

The RAQ consisted of four sections exploring the following aspects in the last three months: (I) computer and Internet access at home and reasons for lack of (4 items); (II) frequency of computer use (2 items); (III) frequency, location, mode and purposes of Internet access or use (5 items); and (IV) computer and Internet skills training and perceived competence (6 items). Questions were constructed in single- or multiple-response multiple-choice formats.

Discussion

We describe the development and validation of the WPROCS-NARA, an instrument that explores three critical sources of opportunities for and barriers to the use of a web-based capture system for patient-reported outcomes among cancer patients—needs, acceptance and readiness.

In assessing the need for WPROCS, it is logical to start with health status. However, health status alone and by itself is not a consistent predictor for need and usage of health information technology. Studies that show increased usage with better health (11) probably reflect better functional ability, while others that show increased usage with poorer health (12,13) probably reflect greater need. High symptom burden was seen among adult Filipino cancer patients with early- and advanced-disease alike (14). Furthermore, among cancer patients, increased usage of online self-management support was seen among patients with low levels of social support and high levels of symptom distress and depression (15). Therefore, we developed the NAQ to explore subjective perception of overall and emotional health, ambulatory status and presence of disabilities; actual needs or illness burden (such as need for assistance in routine and personal care, consultation outside scheduled check-ups, urgent care, and hospitalization); presence of symptoms, symptom distress and symptom medications; and access to healthcare and levels of social support.

Socio-demographic data such as age, gender and educational attainment have not been shown to be consistent predictors of technology acceptance. A systematic review in 2009 showed that age does not have a consistent effect, gender has no effect and higher education has a positive effect on technology acceptance (16). It was suggested that less acceptance with higher age probably reflected lesser computer literacy rate in adults, which may no longer be true today especially in urban populations. On the other hand, some studies have likewise found female gender is associated with greater relative importance of computer anxiety and perceived behavioral control (perception that use of the technology is within the individual’s control and the availability of resources to use it) in technology acceptance (17-22). Indeed, perceived ease of use and perceived usefulness have been postulated to be the main predictors of technology acceptance (Davis’ Technology Acceptance Model) (23). In addition, computer/technology self-efficacy and computer anxiety have been shown to be as reliable predictors for technology acceptance. We therefore used these four factors as basis for developing the AAQ.

Prior exposure to computer/health technology and high levels of computer experience were associated with increased acceptance and usage (15,16). While greater Internet access have been associated with factors such as current employment, higher annual income and higher educational attainment (24), these factors may no longer be as significant as Internet has become increasingly available with the advent of smartphones and mobile data packs. In developing the RAQ, we therefore sought to explore access to information and communication technologies, usage of computers and the Internet, and skills and experience in using the computer and Internet.

The WPROCS-NARA is a valid and reliable tool in understanding the needs, acceptance and readiness among adult cancer patients for a web-based PROs capture system. We have undertaken a survey using this instrument to guide and support future development of a web- and/or short message service (SMS)-based application to facilitate symptom screening and evaluation of symptom management outcomes through remote symptom-reporting. The findings of this survey are reported elsewhere.

Acknowledgments

Funding: None.

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jhmhp.2018.05.03). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). This study was reviewed and approved by the University of Santo Tomas Hospital—Institutional Review Board (IRB-MD-04-2015-054-A1). A written informed consent form was obtained.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Hesse BW, Hanna C, Hesse NK, et al. Outside the box: will information technology be a viable intervention to improve the quality of cancer care? J Natl Cancer Inst Monogr 2010;2010:81-9.
2. Mendel P, Meredith LS, Schoenbaum M, et al. Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research. Adm Policy Ment Health 2008;35:21-37. [Crossref] [PubMed]
3. Holt DT, Helfrich CD, Hall CG, et al. Are you ready? How health professionals can comprehensively conceptualize readiness for change. J Gen Intern Med 2010;25:50-5. [Crossref] [PubMed]
4. Hsu CC, Sandford BA. The Delphi technique: making sense of consensus. Practical Assessment, Research and Evaluation 2007;10:
5. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74. [Crossref] [PubMed]
6. Cody R. Learning SAS by examples: A Programmer’s Guide. Available online: http://support.sas.com/publishing/pubcat/chaps/55129.pdf (access 11 May 2018).
7. Organisation for Economic Co-Operation and Development. Handbook on Constructing Composite Indicators. France: OECD, 2008. Available online: http://www.oecd.org/std/42495745.pdf (accessed 11 May 2018).
8. CAHPS Health Plan Survey. Content last reviewed March 2018. Agency for Healthcare Research and Quality, Rockville, MD. Available online: http://www.ahrq.gov/cahps/surveys-guidance/hp/index.html
9. Morris MG, Venkatesh V. Age difference in technology adoption decisions: implications for a changing workforce. Personnel Psychology 2000;53:375-403. [Crossref]
10. Community survey on ICT usage in households and by individuals 2011 Eurostat model questionnaire (version 3.1). Available online: http://ec.europa.eu/eurostat/ict (accessed 11 May 2018).
11. Chae YM, Park HJ, Cho JG, et al. The reliability and acceptability of telemedicine for patients with schizophrenia in Korea. J Telemed Telecare 2000;6:83-90. [Crossref] [PubMed]
12. Jeannot JG, Froehlich F, Wietlisbach V, et al. Patient use of the Internet for health care information in Switzerland. Swiss Med Wkly 2004;134:307-12. [PubMed]
13. Millard RW, Fintak PA. Use of the Internet by patients with chronic illness. Dis Manag Health Outcomes 2002;10:187-94. [Crossref]
14. Bacorro WR, Que JC, Sy Ortin TT, et al. A cross-sectional analysis of symptom burden among adult cancer patients in a Filipino tertiary care cancer center. J Clin Oncol 2015;33:abstr 98.
15. Børøsund E, Cvancarova M, Ruland CM, et al. How User Characteristics Affect Use Patterns in Web-Based Illness Management Support for Patients with Breast and Prostate Cancer. J Med Internet Res 2013;15:e34 [Crossref] [PubMed]
16. Or CK, Karsh BT. A systematic review of patient acceptance of consumer health information technology. J Am Med Inform Assoc 2009;16:550-60. [Crossref] [PubMed]
17. Gilroy FD, Desai HB. Computer anxiety: Sex, race, and age. Int J Man Machine Stud 1986;25:711-9. [Crossref]
18. Igbaria M, Chakrabarti A. Computer anxiety and attitudes towards microcomputer use. Behav Inf Technol 1990;9:229-41. [Crossref]
19. Venkatesh V, Morris MG. Why don’t men ever stop to ask for directions? Gender, social influence, and their role in technology acceptance and usage behavior. MIS Q 2000;24:115-39. [Crossref]
20. Venkatesh V, Morris MG, Ackerman PL. A longitudinal field investigation of gender differences in individual technology adoption decision-making processes. Organ Behav Hum Decis Process 2000;83:33-60. [Crossref] [PubMed]
21. Taylor S, Todd PA. Understanding information technology usage: A test of competing models. Inf Syst Res 1995;6:144-76. [Crossref]
22. Mathieson K. Predicting user intentions: Comparing the technology acceptance model with the theory of planned behavior. Inf Syst Res 1991;2:173-91. [Crossref]
23. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989;13:319. [Crossref]
24. Abdullah M, Theobald DE, Butler D, et al. Access to communications technologies in a sample of cancer patients: an urban and rural survey. BMC Cancer 2005;5:18. [Crossref] [PubMed]