Challenges in the implementation of electronic systems for patient report of symptoms in oncology: a scoping review

Introduction

Under-recognition and under-treatment of symptoms is a significant problem for patients (1-4). The cost of undertreated symptoms is substantial, in terms of increased suffering, decreased compliance with treatment regimens, and additional interactions with the healthcare system (5-13). Traditionally, symptoms are captured by clinicians when they document clinical notes in the electronic medical record following patient interactions (14,15). This documentation of symptoms is inconsistent, difficult to retrieve, and often incomplete, thus limiting our ability to examine symptom trends (16-20).

Self-report of systems through patient-reported outcome (PRO) instruments is now considered the gold standard for symptom assessment (21-24). Indeed, a growing body of literature demonstrates that the use of PROs can enhance patient and clinician communication, inform health decision-making, and improve symptom management (25-31). For pragmatic and regulatory reasons, electronic systems provide the best method for collecting PROs (25). Fortunately, numerous electronic systems have been developed that enable patients to self-report their own symptoms (25,32,33). Many of these systems were developed specifically for patients with cancer. Because of the extensive efforts and success in developing oncology-focused symptom reporting systems, and because the implementation of electronic symptom reporting is most effective when tailored to address the needs of a specific clinical setting (34-37), this review focuses on patients with cancer undergoing treatment in the United States.

Research has already demonstrated that electronic symptom reporting systems may mitigate under-treatment of symptoms and improve patient outcomes (25,28,33,38-41). For patients with cancer, these electronic systems provide valuable insights into the variable nature of symptom trajectories throughout cycles of treatment as well as during cancer remission and progression. Despite this evidence, these systems have not been widely integrated into clinical practice and many challenges to their adoption remain, including the financial investment at startup, patients’ unwillingness and/or inability to complete measures, unclear interpretation and use of patient reports of symptoms, and additional burdens to clinicians’ workload and time (25,27,28,32,34-37,42,43). Research efforts are underway to assess the feasibility and efficacy of integrating various electronic symptom reporting systems into clinical practice. Missing from the literature, however, is a comprehensive evaluation of the challenges experienced when attempts were made to implement an electronic symptom reporting system into clinical practice.

The objective of this scoping review was to identify challenges that arose in the implementation of electronic systems for patient report of symptoms in oncology clinical practice over time, and approaches that were taken or recommended to overcome those challenges. For the purposes of this review, an electronic system was defined as an application, device, or website used to collect patient-reported symptom data from patients undergoing treatment for cancer. The results of this review could facilitate the implementation of electronic symptom reporting systems into the oncology clinical setting. A scoping review was chosen because of the intent to summarize findings from primary research studies (44). We present this article in accordance with the PRISMA-ScR reporting checklist (available at http://dx.doi.org/10.21037/jhmhp-20-108) (45).

Methods

This scoping review was conducted using the Arkey and O’Malley methodology, as updated by Levac, Colquhoun, and O’Brien (44,46). The protocol developed for this scoping review was not eligible for submission to PROSPERO (47). However, we searched PROSPERO prior to conducting this review to prevent overlap with existing systematic reviews. Articles were managed and screened using Covidence, an electronic platform that facilitates systematic reviews (48).

Inclusion criteria

This review focused on challenges experienced in the implementation of electronic symptom reporting systems, from the perspective of the patient, clinician, and researcher. Both experimental designs (including randomized controlled trials and quasi-experimental designs) and non-randomized observational studies were included. All included studies were longitudinal and prospective in nature and involved the collection of PROs at two or more time points, due to the intention to consider implementation over time. Systematic reviews, literature reviews, and expert opinions were excluded, as they are not primary sources. In addition, usability, instrument validation, and intervention efficacy studies were excluded, as the focus was on the implementation of electronic systems. Abstracts from conferences were included to reduce publication bias. Articles that were not published in English were excluded from this review, although the electronic systems could incorporate PRO instruments in multiple languages. There were no limits on the age of participants. Studies were included if they collected PRO symptom data from patients who were undergoing treatment for any type of cancer in the United States, as this was the focus population and location chosen for this review. Hence, studies of cancer survivors and caregivers of patients with cancer were excluded. Given the rapidly evolving nature of technology, we limited studies to those published from January 2005 to May 2020.

Search strategy

A sensitive search strategy was crafted with the assistance of a medical librarian. Search terms were crafted using controlled vocabulary and keywords specific to each database using the following concepts: patient-reported outcome, symptom assessment, cancer, and electronic or web-based. Publication date restrictions were used in the database searches. The complete search strategy is provided in Table 1. The following electronic databases were searched for relevant studies:

• Cochrane Central Register of Controlled Trials (searched May 26, 2020)
• MEDLINE via Ovid SP (from January 1, 2005 to May 26, 2020)
• CINAHL from EBSCO host (from January 1, 2005 to May 26, 2020).

After articles were identified in the aforementioned searches, duplicates were removed. Two authors then independently screened the titles and abstracts of each article to eliminate articles that were clearly irrelevant. A third, senior author independently resolved any conflicts in the title and abstract screening, to determine if the disputed articles would progress to the next stage of the review, during which two authors independently reviewed the full text of the articles to determine their eligibility for inclusion. In the case of a disagreement, the full text article in question was reviewed by the team and the decision to include or exclude the article was determined by consensus. Reasons for exclusion were recorded in Covidence.

The reference lists of all included studies were manually reviewed by KG and EP to identify articles missed in the database searches. In addition, the names of electronic symptom reporting systems discussed in previous reviews by Jensen et al. (32), Warrington et al. (33), and Meirte et al. (25) were also searched in PubMed and Google Scholar to identify related studies meeting inclusion criteria that were not otherwise identified. For conference abstracts that met the inclusion criteria, the authors specifically looked for associated publications. If a related publication was identified, the conference abstract was excluded to defer to publications, which had more complete reports of the findings.

After completion of all searches and screening, a data extraction plan was developed and then independently pilot tested by all the authors. Adjustments to the data extraction plan were made by consensus before data extraction began. Authors extracted data from the included articles independently using Atlas.ti (cloud version). As this was a scoping review that encompassed a wide variety of study designs, a formal quality assessment of the studies was not performed and the evidence was not evaluated, but rather limitations of the articles were noted during data extraction (44,46). The authors of the articles were not contacted if specific information was not included in the manuscript.

For each of the included studies, the following were extracted in Atlas.ti (cloud version): study design, funding source/sponsor, inclusion/exclusion criteria, recruiting process, timing of recruitment and data collection, and participant attrition. For the participants of the included studies, we sought to record their age, cancer type, access to technology, education level, gender, race/ethnicity, and differences in baseline characteristics for studies that compared two or more groups. In terms of the electronic system, we collected the system name, system features, system design, the PRO instruments used, the PRO completion process, setting of PRO completion (clinic, home, hospital, etc.), languages offered, participant preparation or system training, and any co-interventions. Last, we extracted challenges experienced with the electronic system from the perspective of participants, clinicians, and researchers and their satisfaction with the system, number of alerts, reported actions by clinicians on alerts, participant attrition, usability assessments, limitations of the studies, any correlation of PROs to clinician assessments, and overall lessons learned from each study.

Extracted data were collated and summarized. Using Braun et al.’s method (49), SC and KG then conducted an inductive thematic analysis of the challenges implementing the electronic symptom reporting systems, from the perspective of the patient, clinician, and researcher (i.e., themes were extracted from the data and not pre-identified).

Results

The search of the electronic databases yielded 3,133 citations when queried on May 26, 2020. After de-duplication with a reference manager, a total of 2,457 citations were reviewed for relevance by two reviewers. Screening the titles and abstracts of those records identified 99 studies as potentially relevant, and the full-text publications were obtained for a more detailed review. After reviewing full texts of potential studies and reviewing the bibliographies of the included studies, 20 research studies met the inclusion criteria and were included in the final narrative scoping review. The PRISMA flowchart of study selection is presented in Figure 1.

Figure 1 PRISMA diagram of study selection.

Of all the included studies, sixteen were supported by NIH funding, institutional grants, or other non-profit grant funding, two were supported by industry (Pfizer or SOS Inc. or an insurance company), and two did not report funding sources. All were prospective research studies. Fourteen had one arm and were conducted at a single institution and the remaining six had two arms and were conducted through multiple institutions, including three studies conducted at more than 35 institutions. The two-armed studies involved comparison of: (I) patient report of symptoms via an electronic system versus usual care (symptom reporting directly to a clinician) (38), (II) frequent or unlimited access to an electronic symptom reporting system versus access only at limited time points (50,51), (III) both arms having similar access to an electronic symptom reporting system but with patients and/or clinicians receiving compiled symptom reports in one arm only (51-53), and (IV) both arms having similar access to an electronic symptom reporting system but with patients receiving reminders/prompts to use the system in one arm only (52,54). The overwhelming majority of articles found focused on medical oncology, likely because patients are followed over a longer period and medical oncology has a greater focus on symptom management. The study characteristics table (https://cdn.amegroups.cn/static/public/jhmhp-20-108-1.pdf) lists key characteristics of the included studies (38,50-68).

The electronic systems for symptom reporting employed in the included articles were completed by patients in the waiting room prior to clinic appointments, during hospital admissions, or at home. Five articles described an implementation of the STAR (Symptom Tracking and Reporting) system, four articles described an implementation of the PRO-CTCAE (Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events), and the remaining articles report on other various systems. All the systems reminded or prompted the patient when it was time to login and complete a symptom report, although the nature of the reminder/prompt varied. Other details of the systems’ features are provided in the system features table (https://cdn.amegroups.cn/static/public/jhmhp-20-108-2.pdf). Of the articles included in this review, ten reported some alert capabilities. Most automatically notified the clinical team via email (38,52,54,56,57,60,62,65), one required the study staff to review symptom reports and personally email the clinical team (68), and seven alerted the patient to contact the clinical team for follow-up (38,39,50,56,57,60,65). An overwhelming majority of the articles reported patient satisfaction with the systems and positive assessments of ease of understanding and usefulness. Overall, clinicians also had a positive view of the systems and thought they were helpful for symptom management and potentially useful for future research, although some clinicians noted that patients rated symptoms more severely than clinicians (56,62). In one article, clinicians specifically reported that the system helped them identify areas of concern for patients that otherwise might have gone unnoticed (65). However, in other articles, clinicians questioned the accuracy of the patient assessments, felt that system alerts were often unwarranted, or found that that PRO data impacted the care they provided to patients (62,65).

Nine categories of challenges implementing electronic symptom reporting systems were identified through the thematic analysis (see Table 2 for the list of categories). Challenges were categorized as: (I) technical issues, (II) system usability issues, (III) patient lack of comfort with or knowledge of technology, (IV) incomplete/missing data, (V) lack of patient use of the system, (VI) other patient issues, (VII) difficulties timing completion with clinical processes, (VIII) lack of clinic staff involvement/engagement, and (IX) lack of clinician comfort/knowledge regarding how to use PRO data. The challenges and associated strategies table (https://cdn.amegroups.cn/static/public/jhmhp-20-108-3.pdf) lists the types of challenges identified in each article, as well as the strategies suggested or attempted by the authors to overcome those challenges. Although the frequency with which challenges were reported are tabulated in the challenges and associated strategies table (https://cdn.amegroups.cn/static/public/jhmhp-20-108-3.pdf), this numerical data should be interpreted with caution as only challenges that were discussed in the articles are included. Other challenges were likely experienced but were not discussed in the articles due to word length limitations and the authors’ selected focus.

Patients most often reported technical issues (57,58,67), which included problems accessing the internet and infrequent usage of email (58,61,63). In addition, patients reported that reminders from clinic staff to log in and report symptoms were necessary to encourage use of the system (59,60) and expressed concerns that the system limited direct communication with the health care team (62,65). Similarly, clinicians most often reported technical issues when accessing the system (55,62,65,66). In certain articles, clinicians additionally reported having difficulty interpreting the PRO reports (65) and expressed doubt regarding the value of electronic symptom reporting systems when their use did not substantively impact their treatment plans for patients (64).

While not one of the most frequent issues, lack of time was noted as a potentially significant barrier to use of electronic symptom reporting systems by both patients and clinicians. Several authors reported that completing the symptom assessments did not require much time from patients (54,55,57). However, lack of time seems to reflect the need to incorporate another task—use of the electronic system—into clinicians’ and patients’ already busy schedules. Since the included articles were research studies, the systems were tested with the support of dedicated research staff and processes, which may impact the sustainability of the electronic symptom reporting systems after the studies ended, as was noted in some articles (55,60).

A variety of potential strategies were proposed to overcome the challenges identified in the articles (see https://cdn.amegroups.cn/static/public/jhmhp-20-108-3.pdf). While the proposed strategies were based on participants’ and clinicians’ experiences during the research studies, they had not yet been tested in practice to evaluate their effectiveness. The proposed strategies focused on improvements in three areas: (I) system design, (II) integration with clinical flow/patient care, and (III) system implementation. Suggested system design improvements included: ensuring clinicians and patients were included throughout the system design process and conducted usability testing; offering both internet-based and downloadable applications to access the system; adapting the PRO data collected based on the type of cancer diagnosis and its severity to minimize patient burden; and, selecting a system that provides features like automated reminders to complete symptom assessments, clinician alerts for severe symptoms, symptom management guidance, and the ability to report symptoms between hospital/clinic visits and on any device (38,50,52,54,57-59,61-63,65,68). Suggested improvements to integration with clinical flow/patient care included: using clinician champions to facilitate patients’ completion of PRO symptom assessments and clinicians’ use of PRO data, and developing and disseminating a clear plan to add electronic symptom reporting into the flow of routine clinical care for both patients and clinicians (55,60,61,65,67,68). Suggested improvements to system implementation included: ensuring wireless internet signal strength throughout the hospital/clinic setting, providing a brief computerized tutorial on the system and its implementation into the clinical setting, and having technical support available to address issues that arise with the system.

Discussion

The objective of this scoping review was to identify challenges experienced and potential strategies to facilitate the implementation of electronic systems for patient report of symptoms in oncology clinical practice. Experts have posited that the challenges to integrating these systems into clinical practice include the financial investment at startup, patient unwillingness or inability to complete measures, unclear interpretation and use of patient reports of symptoms, and additional burdens to clinicians’ workload and time (25,28,32,34-37,42,43). Although this review confirmed many of these challenges, it also revealed an overall patient and clinician satisfaction with the use of electronic symptom reporting systems.

Implementing an electronic symptom reporting system necessitates a significant investment into the system. Administrators must carefully consider the clinical environment, the patient population and needs, and resources available to maintain the system when selecting an electronic system. System usability and technical issues go hand in hand; when a system is more usable, fewer technical issues are likely to arise. “Usable” is further defined by the Task, User, Representation, and Function (TURF) framework as “easy to learn, easy to use, and error-tolerant” (69). Poorly designed health technology that fails to consider users’ needs often has unintended negative effects on efficiency, user satisfaction, and health care quality (70-73). The technical issues reported in this review would have adversely impacted the experience of patients and clinicians with those electronic symptom reporting systems, increased the prevalence of missing PRO data, and prevented the successful integration of those systems into clinical practice.

Efforts need to be directed to ensuring that clinicians understand how patient-reported symptoms can be used to enhance clinical practice. Clinicians may be unsure how to address patient-reported symptoms and may need education about PRO item content and the meaning of PRO scores (65,74). Methods proposed to enhance clinician comfort and knowledge of PRO symptom data include the following. Clinician panels can develop clinical decision support protocols for responding to PRO symptom data, and these protocols can be readily posted within the electronic system (65). Visualization of PRO symptom reports should be standardized and graphic-driven so that it is clear whether a higher score indicates a better or worse symptom (65). Clinicians will likely require time and experience with PRO data before they feel comfortable interpreting and effectively using patient symptom reports to guide the patient’s clinical care.

Many of the challenges and strategies that were identified in this review support previous findings in the literature. One unanticipated finding by Snyder et al. (65) and Tran et al. (67), was that patients with advanced or late-stage cancer were less willing to report symptoms as their disease progressed. For patients who have a terminal diagnosis, reporting symptoms and viewing the symptom reports emphasized what they had lost and would never recover (65,67). Further research is needed to consider how patient-reported symptom data should be used in this population. These results emphasize the need to tailor the PRO data collected for the specific patient population and clinical setting. Certain instruments or questions may only be appropriate in certain contexts. For example, patients with a terminal illness may benefit the most from PRO instruments that focus on psychosocial and spiritual quality of life and treatable physical symptoms. Item-response theory and computer adaptive testing are facilitating these efforts and are being used with greater prevalence to develop PRO systems that tailor the questions asked to the specific patient population (75,76).

Ensuring accessibility is key to the successful implementation of electronic symptom reporting systems, particularly in the clinical setting. One significant gap in the identified studies was the lack of discussion regarding how the electronic systems have been/or should be adapted for patients with disabilities, so that patients who are visually impaired, have decreased hand mobility, or have other disabilities may also report symptoms. Further research is needed to explore the accessibility of these electronic systems.

Another aspect of accessibility that is important to acknowledge in this review is technological literacy. There is a “digital divide” in the United States between those who have access to and are comfortable with technology versus those who are not. Rather than contributing to this divide, successful implementation of electronic symptom reporting systems will allow all participants to access and use the system, regardless of prior experience with technology. Only one of the articles considered the level of prior computer experience (38). Basch et al. found that the use of an electronic system for symptom reporting had a greater impact on the health outcomes of computer-inexperienced participants, perhaps because these participants benefited the most when using a structured method to report symptoms (38). When implementing an electronic symptom reporting system, administrators need to ensure that patients with less computer experience are not deterred from using the system, by a lack of access to a computer or by a lack of knowledge of technology.

Limitations

This scoping review had several limitations. First, publication bias should be considered in all reviews, as studies that demonstrate positive outcomes are more likely to be published. This review did not undertake a formal assessment of publication bias. Another limitation of this review is that the search was limited to systems implemented in the United States and studies published in English. Health care varies widely based on the context in which it is delivered; thus, challenges identified in research from other countries or in other languages may have been overlooked. Another limitation of this review is that no formal risk of bias assessment was completed for the included studies, although relevant limitations were noted by the authors when reviewing included articles. As this review centered on prospective interventional studies that demonstrated implementation of these systems, challenges identified through qualitative descriptive studies may have been missed. Though future work may focus on the nuance of the challenges for specific cancer types, in this review we sought to describe the state of the science around the implementation of electronic symptom reporting systems for patients with cancer in general.

Conclusions

Electronic symptom reporting systems could significantly improve clinicians’ ability to identify and thus treat patients’ symptoms. These improvements will only be possible if these systems can be successfully implemented into clinical practice. Unfortunately, PROs are still not routinely being used in the care of patients with cancer (27). The findings of this review highlight the challenges that need to be addressed when implementing an electronic symptom reporting system for patients with cancer and potential strategies for overcoming these challenges. There is no one solution that will overcome all the challenges to implementing electronic systems into clinical practice; approaches need to be tailored for the specific patient population and clinical setting (26). This review could help hospital administrators and clinicians prepare for and improve the implementation of electronic symptom reporting systems in practice.

Acknowledgments

Funding: This work was generously supported in part by a Scholar Award from the St. Baldrick’s Foundation, by a Summer Research Experience Grant from the Cancer Prevention and Research Institute of Texas, and by the Mentored Patient-Oriented Research Career Development Award – “Facilitating Comprehensive Self and Proxy Symptom Assessments for Children with Cancer” (K23 NR019294), National Institute of Nursing Research.

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editors (Naleef Fareed, Ann Scheck McAlearney, and Susan D Moffatt-Bruce) for the series “Innovations and Practices that Influence Patient-Centered Health Care Delivery” published in Journal of Hospital Management and Health Policy. The article has undergone external peer review.

Reporting Checklist: The authors have completed the PRISMA-ScR reporting checklist. Available at http://dx.doi.org/10.21037/jhmhp-20-108

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jhmhp-20-108). The series “Innovations and Practices that Influence Patient-Centered Health Care Delivery” was commissioned by the editorial office without any funding or sponsorship. Dr. Crane reports grants from Cancer Prevention and Research Institute of Texas and from St. Baldrick’s Foundation, during the conduct of the study. Dr. Nosich reports a grant from Cancer Prevention and Research Institute of Texas, during the conduct of the study. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Kavalieratos D, Gelfman LP, Tycon LE, et al. Palliative care in heart failure: Rationale, evidence, and future priorities. J Am Coll Cardiol 2017;70:1919-30. [Crossref] [PubMed]
2. Robinson MT, Holloway RG. Palliative Care in Neurology. Mayo Clin Proc 2017;92:1592-601. [Crossref] [PubMed]
3. van den Beuken-van Everdingen MH, Hochstenbach LM, Joosten EA, et al. Update on prevalence of pain in patients with cancer: Systematic review and meta-analysis. J Pain Symptom Manage 2016;51:1070-90. [Crossref] [PubMed]
4. Alexander K, Goldberg J, Korc-Grodzicki B. Palliative care and symptom management in older patients with cancer. Clin Geriatr Med 2016;32:45-62. [Crossref] [PubMed]
5. Yang P, Cheville AL, Wampfler JA, et al. Quality of life and symptom burden among long-term lung cancer survivors. J Thorac Oncol 2012;7:64-70. [Crossref] [PubMed]
6. Verkissen MN, Hjermstad MJ, Van Belle S, et al. Quality of life and symptom intensity over time in people with cancer receiving palliative care: Results from the international European Palliative Care Cancer Symptom study. PLoS One 2019;14:e0222988. [Crossref] [PubMed]
7. Paque K, Elseviers M, Vander Stichele R, et al. Changes in medication use in a cohort of patients with advanced cancer: The international multicentre prospective European Palliative Care Cancer Symptom study. Palliat Med 2018;32:775-85. [Crossref] [PubMed]
8. Kaufmann TL, Rendle KA, Aakhus E, et al. Views from patients with cancer in the setting of unplanned acute care: Informing approaches to reduce health care utilization. JCO Oncol Pract 2020;16:e1291-e1303. [Crossref] [PubMed]
9. Deshields TL, Potter P, Olsen S, et al. The persistence of symptom burden: symptom experience and quality of life of cancer patients across one year. Support Care Cancer 2014;22:1089-96. [Crossref] [PubMed]
10. Yue X, Pruemer JM, Hincapie AL, et al. Economic burden and treatment patterns of gynecologic cancers in the United States: evidence from the Medical Expenditure Panel Survey 2007-2014. J Gynecol Oncol 2020;31:e52. [Crossref] [PubMed]
11. Deshields TL, Penalba V, Liu J, et al. Comparing the symptom experience of cancer patients and non-cancer patients. Support Care Cancer 2017;25:1103-9. [Crossref] [PubMed]
12. Jacobs JM, Pensak NA, Sporn NJ, et al. Treatment satisfaction and adherence to oral chemotherapy in patients with cancer. J Oncol Pract 2017;13:e474-85. [Crossref] [PubMed]
13. Jacobs JM, Ream ME, Pensak N, et al. Patient experiences with oral chemotherapy: Adherence, symptoms, and quality of life. J Natl Compr Canc Netw 2019;17:221-8. [Crossref] [PubMed]
14. Gutheil TG. Fundamentals of medical record documentation. Psychiatry (Edgmont) 2004;1:26-8. [PubMed]
15. Mathioudakis A, Rousalova I, Gagnat AA, et al. How to keep good clinical records. Breathe (Sheff) 2016;12:369-73. [Crossref] [PubMed]
16. O’Donnell HC, Kaushal R, Barrón Y, et al. Physicians’ attitudes towards copy and pasting in electronic note writing. J Gen Intern Med 2009;24:63-8. [Crossref] [PubMed]
17. Weis JM, Levy PC. Copy, paste, and cloned notes in electronic health records. Chest 2014;145:632-8. [Crossref]
18. Yadav S, Kazanji N. Comparison of accuracy of physical examination findings in initial progress notes between paper charts and a newly implemented electronic health record. J Am Med Inform Assoc 2017;24:140-4. [Crossref] [PubMed]
19. Bowman S. Impact of electronic health record systems on information integrity: Quality and safety implications. Perspect Health Inf Manag 2013;10:1c. [PubMed]
20. Sittig DF, Singh H. Defining health information technology-related errors: New developments since to err is human. Arch Intern Med 2011;171:1281-4. [Crossref] [PubMed]
21. Leahy AB, Feudtner C, Basch E. Symptom monitoring in pediatric oncology using patient-reported outcomes: Why, how, and where next. Patient 2018;11:147-53. [Crossref] [PubMed]
22. Calvert M, Kyte D, Price G, et al. Maximising the impact of patient reported outcome assessment for patients and society. BMJ 2019;364:k5267. [Crossref] [PubMed]
23. Mooney K, Berry DL, Whisenant M, et al. Improving cancer care through the patient experience: How to use patient-reported outcomes in clinical practice. Am Soc Clin Oncol Educ Book 2017;37:695-704. [Crossref] [PubMed]
24. Fares CM, Williamson TJ, Theisen MK, et al. Low Concordance of patient-reported outcomes with clinical and clinical trial documentation. JCO Clin Cancer Inform 2018;2:1-12. [Crossref] [PubMed]
25. Meirte J, Hellemans N, Anthonissen M, et al. Benefits and disadvantages of electronic patient-reported outcome measures: Systematic review. JMIR Perioper Med 2020;3:e15588. [Crossref] [PubMed]
26. Stover AM, Haverman L, van Oers HA, et al. Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res 2020; Epub ahead of print. [Crossref] [PubMed]
27. Stover AM, Stricker CT, Hammelef K, et al. Using stakeholder engagement to overcome barriers to implementing patient-reported outcomes (PROs) in cancer care delivery: Approaches from 3 prospective studies. Med Care 2019;57:S92-9. [Crossref] [PubMed]
28. Aapro M, Bossi P, Dasari A, et al. Digital health for optimal supportive care in oncology: Benefits, limits, and future perspectives. Support Care Cancer 2020;28:4589-4612. [Crossref] [PubMed]
29. Fiteni F, Cuenant A, Favier M, et al. Clinical relevance of routine monitoring of patient-reported outcomes versus clinician-reported outcomes in oncology. In Vivo 2019;33:17-21. [Crossref] [PubMed]
30. Graupner C, Kimman ML, Mul S, et al. Patient outcomes, patient experiences and process indicators associated with the routine use of patient-reported outcome measures (PROMs) in cancer care: a systematic review. Support Care Cancer 2021;29:573-93. [Crossref] [PubMed]
31. Penedo FJ, Oswald LB, Kronenfeld JP, et al. The increasing value of eHealth in the delivery of patient-centred cancer care. Lancet Oncol 2020;21:e240-51. [Crossref] [PubMed]
32. Jensen RE, Snyder CF, Abernethy AP, et al. Review of electronic patient-reported outcomes systems used in cancer clinical care. J Oncol Pract 2014;10:e215-22. [Crossref] [PubMed]
33. Warrington L, Absolom K, Conner M, et al. Electronic systems for patients to report and manage side effects of cancer treatment: Systematic review. J Med Internet Res 2019;21:e10875. [Crossref] [PubMed]
34. Chan EK, Edwards TC, Haywood K, et al. Implementing patient-reported outcome measures in clinical practice: A companion guide to the ISOQOL user’s guide. Qual Life Res 2019;28:621-7. [Crossref] [PubMed]
35. Aaronson N, Elliott T, Greenhalgh J, et al. User’s guide to implementing patient-reported outcomes assessment in clinical practice (Version 2). International Society for Quality of Life Research 2015. Available online: https://www.isoqol.org/wp-content/uploads/2019/09/2015UsersGuide-Version2.pdf
36. Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res 2012;21:1305-14. [Crossref] [PubMed]
37. Van Der Wees PJ, Nijhuis‐Van Der Sanden MW, Ayanian JZ, et al. Integrating the use of patient‐reported outcomes for both clinical practice and performance measurement: views of experts from 3 countries. Milbank Q 2014;92:754-75. [Crossref] [PubMed]
38. Basch E, Deal AM, Kris MG, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: A randomized controlled trial. J Clin Oncol 2016;34:557-65. [Crossref] [PubMed]
39. Basch E, Deal AM, Dueck AC, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 2017;318:197-8. [Crossref] [PubMed]
40. Denis F, Basch E, Septans A-L, et al. Two-year survival comparing web-based symptom monitoring vs routine surveillance following treatment for lung cancer. JAMA 2019;321:306-7. [Crossref] [PubMed]
41. Denis F, Lethrosne C, Pourel N, et al. Randomized trial comparing a web-mediated follow-up with routine surveillance in lung cancer patients. J Natl Cancer Inst 2017;109. [Crossref] [PubMed]
42. Warsame R, D’Souza A. Patient reported outcomes have arrived: A practical overview for clinicians in using patient reported outcomes in oncology. Mayo Clin Proc 2019;94:2291-2301. [Crossref] [PubMed]
43. Jim HSL, Hoogland AI, Brownstein NC, et al. Innovations in research and clinical care using patient-generated health data. CA Cancer J Clin 2020;70:182-199. [Crossref] [PubMed]
44. Arksey H, O'Malley L. Scoping studies: Towards a methodological framework. Int J Soc Res Methodol 2005;8:19-32. [Crossref]
45. Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med 2018;169:467-73. [Crossref] [PubMed]
46. Levac D, Colquhoun H, O'Brien KK. Scoping studies: Advancing the methodology. Implementation Science 2010;5:69. [Crossref] [PubMed]
47. NIHR Centre for Reviews and Dissemination at York. PROSPERO: International prospective register of systematic reviews. Available online: https://www.crd.york.ac.uk/PROSPERO/.
48. Covidence. Better systematic review management. Available online: https://www.covidence.org/.
49. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006;3:77-101. [Crossref]
50. Berry DL, Hong F, Halpenny B, et al. Electronic self-report assessment for cancer and self-care support: Results of a multicenter randomized trial. J Clin Oncol 2014;32:199-205. [Crossref] [PubMed]
51. Bryant AL, Coffman E, Phillips B, et al. Pilot randomized trial of an electronic symptom monitoring and reporting intervention for hospitalized adults undergoing hematopoietic stem cell transplantation. Support Care Cancer 2020;28:1223-31. [Crossref] [PubMed]
52. Wolfe J, Orellana L, Cook EF, et al. Improving the care of children with advanced cancer by using an electronic patient-reported feedback intervention: Results from the PediQUEST randomized controlled trial. J Clin Oncol 2014;32:1119-26. [Crossref] [PubMed]
53. Fromme EK, Holliday EB, Nail LM, et al. Computerized patient-reported symptom assessment in radiotherapy: a pilot randomized, controlled trial. Support Care Cancer 2016;24:1897-906. [Crossref] [PubMed]
54. Graetz I, McKillop CN, Stepanski E, et al. Use of a web-based app to improve breast cancer symptom management and adherence for aromatase inhibitors: a randomized controlled feasibility trial. J Cancer Surviv 2018;12:431-40. [Crossref] [PubMed]
55. Abernethy AP, Herndon JE, Wheeler JL, et al. Feasibility and acceptability to patients of a longitudinal system for evaluating cancer-related symptoms and quality of life: Pilot study of an e/Tablet data-collection system in academic oncology. J Pain Symptom Manage 2009;37:1027-38. [Crossref] [PubMed]
56. Andikyan V, Rezk Y, Einstein MH, et al. A prospective study of the feasibility and acceptability of a Web-based, electronic patient-reported outcome system in assessing patient recovery after major gynecologic cancer surgery. Gynecol Oncol 2012;127:273-7. [Crossref] [PubMed]
57. Basch E, Artz D, Dulko D, et al. Patient online self-reporting of toxicity symptoms during chemotherapy. J Clin Oncol 2005;23:3552-61. [Crossref] [PubMed]
58. Basch E, Dueck AC, Rogak LJ, et al. Feasibility assessment of patient reporting of symptomatic adverse events in multicenter cancer clinical trials. JAMA Oncol 2017;3:1043-50. [Crossref] [PubMed]
59. Basch E, Dueck AC, Rogak LJ, et al. Feasibility of implementing the Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events in a multicenter trial: NCCTG N1048. J Clin Oncol 2018; Epub ahead of print. [Crossref] [PubMed]
60. Basch E, Iasonos A, Barz A, et al. Long-term toxicity monitoring via electronic patient-reported outcomes in patients receiving chemotherapy. J Clin Oncol 2007;25:5374-80. [Crossref] [PubMed]
61. Basch E, Pugh SL, Dueck AC, et al. Feasibility of patient reporting of symptomatic adverse events via the patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE) in a chemoradiotherapy cooperative group multicenter clinical trial. Int J Radiat Oncol Biol Phys 2017;98:409-18. [Crossref] [PubMed]
62. Cowan RA, Suidan RS, Andikyan V, et al. Electronic patient-reported outcomes from home in patients recovering from major gynecologic cancer surgery: A prospective study measuring symptoms and health-related quality of life. Gynecol Oncol 2016;143:362-6. [Crossref] [PubMed]
63. Lucas SM, Kim TK, Ghani KR, et al. Establishment of a web-based system for collection of patient-reported outcomes after radical prostatectomy in a statewide quality improvement collaborative. Urology 2017;107:96-102. [Crossref] [PubMed]
64. McCleary NJ, Wigler D, Berry D, et al. Feasibility of computer-based self-administered cancer-specific geriatric assessment in older patients with gastrointestinal malignancy. Oncologist 2013;18:64-72. [Crossref] [PubMed]
65. Snyder CF, Blackford AL, Wolff AC, et al. Feasibility and value of PatientViewpoint: A web system for patient‐reported outcomes assessment in clinical practice. Psychooncology 2013;22:895-901. [Crossref] [PubMed]
66. Stukenborg GJ, Blackhall L, Harrison J, et al. Cancer patient-reported outcomes assessment using wireless touch screen tablet computers. Qual Life Res 2014;23:1603-7. [Crossref] [PubMed]
67. Tran C, Dicker A, Leiby B, et al. Utilizing digital health to collect electronic patient-reported outcomes in prostate cancer: Single-arm pilot trial. J Med Internet Res 2020;22:e12689. [Crossref] [PubMed]
68. Wright AA, Raman N, Staples P, et al. The HOPE pilot study: Harnessing patient-reported outcomes and biometric data to enhance cancer care. JCO Clin Cancer Inform 2018;2:1-12. [Crossref] [PubMed]
69. Zhang J, Walji MF. TURF: Toward a unified framework of EHR usability. J Biomed Inform 2011;44:1056-67. [Crossref] [PubMed]
70. Ash JS, Sittig DF, Poon EG, et al. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007;14:415-23. [Crossref] [PubMed]
71. Chen Y, editor. Documenting transitional information in EMR. CHI Conference on Human Factors in Computing Systems; 2010 April 10-15; Atlanta, GA: ACM.
72. Evans DC, Nichol WP, Perlin JB. Effect of the implementation of an enterprise-wide Electronic Health Record on productivity in the Veterans Health Administration. Health Econ Policy Law 2006;1:163-9. [Crossref] [PubMed]
73. Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inform 2005;38:75-87. [Crossref] [PubMed]
74. Yang LY, Manhas DS, Howard AF, et al. Patient-reported outcome use in oncology: A systematic review of the impact on patient-clinician communication. Support Care Cancer 2018;26:41-60. [Crossref] [PubMed]
75. Fayers PM. Applying item response theory and computer adaptive testing: the challenges for health outcomes assessment. Qual Life Res 2007;16:187-94. [Crossref] [PubMed]
76. Morris S, Bass M, Lee M, et al. Advancing the efficiency and efficacy of patient reported outcomes with multivariate computer adaptive testing. J Am Med Inform Assoc 2017;24:897-902. [Crossref] [PubMed]