A patient-reported outcome (PRO) tool is “a measurement of any aspect of a patient's health status that comes directly from the patient without being altered by others.”1 An important goal of PROs is to collect data from the patient, unfiltered by clinician interpretation.2 Several early proof-of-concept studies motivated subsequent development and validation of multidimensional PRO tools for assessing the patient's perspective; this was initially in the context of endpoints on prospective trials including adverse events,3 and more recently as an intervention in and of themselves.4

With PRO use dating to 1970s, their initial mode of delivery was paper assessments that patients would complete at the time of the clinic visit. However, with technological advances and widespread adoption of electronic medical records, electronic PROs (ePROs) have become increasingly used, which include either of 2 platforms: voice/auditory reporting or screen-based reporting. Voice/auditory reporting is often telephone-based and commonly referred to as interactive voice response (IVR).5 Alternatively, screen-based reporting systems collect responses in visual text format by using mobile SMS text solutions, computers, or touchscreen tablets.5 Each mode of administration has benefits and drawbacks. Paper-based PROs allow patients to complete the surveys without the need of technical support, internet connectivity, or digital literacy, but necessitate manual review and data entry. Meanwhile, the migration from paper-based to ePROs has facilitated real-time data collection and use, reduced labor costs and data entry errors, but increased informatics costs and introduced digital literacy barriers.6,7

In the migration to ePROs, it was also important to demonstrate that scoring remains consistent and free of bias. “Measurement of equivalence” refers to the concept that a given individual should acquire the same responses and scores when using paper-based PROs and ePROs.5 Meanwhile, “comparability” is determined by the extent to which the original content and format of the paper-based PRO must be modified while transitioning to ePROs.5 Importantly, a study by Norquist et al8 compared paper-based and electronic versions of the EORTC QOL module for patients with head-and-neck cancer and found no differences in patient responses or interpretation of items based on the mode of administration. Two additional systematic reviews support these findings,9,10 endorsing the equivalence of ePROs versus paper-based PROs and transitioning to electronic versions being preferred.11,12 Nonetheless, ePROs pose potentially greater upfront informatics challenges and costs, as well as potential patient barriers, leading multi-institutional prospective trials to still commonly collect paper-based surveys.

The development and validation of PROs is important for establishing them as accurate and appropriate reporting tools, both clinically and for research purposes. While an in-depth discussion of each step is beyond the scope of this review, a detailed systematic approach is outlined by Rothrock et al,13 and herein, we highlight a few key considerations in the process.

Before developing a PRO instrument, the researcher should be specific about the domain(s) to measure, which can be facilitated by patient interviews and focus groups to understand their experiences and priorities. Thereafter, a comprehensive literature search to determine if an existing questionairre can be used or modified questionairre.13 Modifying existing tools is generally preferred to writing new questions due to the complex nature of PRO tool development. However, this should be done with caution to preserve the accuracy and validity of the tool. Several “best practices” merit consideration regarding item selection. First, ideal items focus on a single endpoint (rather than “double-barreled” questions, ie, a single question about breast tenderness and heaviness) in order to reduce or eliminate confusion for patients.13 Second, the literature has placed a strong emphasis on recall periods, due to the risk of recall bias. According to a study by Mendoza et al,14 the optimal recall period for PRO-CTCAE items in a clinical trial is 7 days. Third, designed items should be accessible to a wide range of audience15 so selecting tools available in multiple languages and modifying minimally is preferred.16 An example of this process is the adaptation of PRO-CTCAE for radiation oncology17

When developing new PRO tools and questionnaires, cognitive interviews are typically conducted in which patients are asked to complete an item and are assessed with a series of questions to measure their understanding of the item along with their thought process in selecting their response.13 The goal is to determine whether the patient's comprehension of the items is similar to that intended by the instrument developer.13 Following item development, the instrument will proceed with initial testing and validation among target populations. If available, responses can be compared with other PRO instrument responses in the same domain.13,16

Validity is the degree to which a PRO instrument measures the domain of interest.18 The International Society of Pharmacoeconomics and Outcomes Research (ISPOR) Good Research Practices Taskforce has proposed detailed recommendations for the validation process.19 Validity can be assessed across 3 dimensions: content validity, construct validity, and criterion validity.20 Following the completion of clinical testing and validation, feedback from patients and clinical trials is collected to promote further improvement in the PRO questionnaire. Clinical validation represents a continuous cyclical process. For example, EORTC QLQ-C30 is a well-known, robustly validated tool that measures health-related quality of life (HRQOL) but is subject to ongoing re-evaluation for improvement in EORTC meetings.

Ultimately, in the context of endpoints on a clinical trial, previously used and validated instruments are preferred, particularly as primary endpoints when a reference for potential effect size is needed. When PROs themselves represent a primary component of the intervention, an even greater need exists for them to be brief and high yield for use in routine practice.

PROs can address a variety of different domains, or topic areas. The following has been adapted from Cella et al.21

Instruments Assessing Overall Health-Related Quality of Life: The EORTC-QLQ-C3022 is a multidimensional 30-item questionnaire that encompasses 5 functional scales, 3 symptom scales, a global health scale, and a quality-of-life scale. It provides an overall picture of an individual's well-being and assesses disease-related symptoms and treatment-related side effects. Malignancy specific EORTC QLQ scales also exist (ie, EORTC QLQ-BR 23 for breast cancer). Similarly, FACT-G23 is a validated and reliable 27-item questionnaire that measures an individual's HRQOL through 5 components: physical, social, family, emotional, and functional well-being. Specific subsets of FACT-G have been validated for use among patients with certain malignancies; for example, FACT-B for breast cancer includes additional components assessing body image, sexual function, and psychological function. A review comparing these scales has been published.24

Instruments Assessing Functional Status: Functional status is defined as a patient's ability to perform basic and advanced activities of daily life.6,25 Although many HRQOL assessments include components of function, perhaps the most unique measure is one that approximates the clinician-reported ECOG performance status, called patient-reported functional status (PRFS). The PRFS is a domain within the Patient-Generated Subjective Global Assessment.26 In a study comparing patient to physician report of performance status for patients with cancer in a palliative care clinic, physicians overestimated the score (0-5 scale) on average 0.31 points, particularly for patients that were younger and with greater symptom burden.27

Instruments Assessing Symptom Burden: Symptom burden scales examine the severity of symptoms (ie, pain, fatigue, anxiety), often alongside their direct impact on functioning or QOL.28 Commonly used symptom PRO tools in radiation oncology clinical trials include but are not limited to MD Anderson Symptom Inventory (MDASI),28,29 Brief Pain Inventory (BPI),28 and Patient-Reported Outcomes Measurement Information System (PROMIS) (which has independent scores for sleep quality,30 pain interference,31 sexual functioning,32 and emotional distress33).

Instruments Assessing Treatment Toxicity: To supplement clinician-reported adverse events in the setting of clinical trials, the National Cancer Institute developed a patient-reported version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE).34

Patient Experience: Increasingly patients are asked about their experience of health care, which can include overall satisfaction via the Press-Ganey survey or the common usability question regarding “likelihood to recommend.”35 Financial toxicity also provides insight into the patient experience and can be assessed using several measures including the FACIT-COST.36

While these categories provide a general framework consideration in PRO selection, they are somewhat fluid. For example, it can be difficult to differentiate disease symptoms from treatment toxicity but considering the purpose of the PRO collection can help identify the most appropriate measure. Additionally, a question about financial toxicity is embedded within QLQ-C30 (Question #28 regarding “Has your physical condition or medical treatment caused you financial difficulties?”) and was used in the recent randomized trial of radiation in metastatic disease by Sahgal et al.37 Meanwhile, the NCCN Distress Thermometer is a psychosocial measure that includes a checklist regarding “Practical Problems” such as housing, childcare, and insurance/financial.

Several considerations exist for identifying and selecting PRO questionnaires for clinical trials. The investigator first needs to decide the general domain(s) they wish to assess (see categories proposed above) and ensure the domain(s) reflect the objective of the study.38 Second, validated and previously used PRO tools are generally preferred on trial for power calculations (especially if used as a primary endpoint) and comparison of results to prior studies. Third, the frequency and timing of PRO assessments should be tailored to the domain being monitored, balancing recall bias with pragmatic elements of the clinical trial including standard data collection points for other endpoints. Often 3-4 time points, including baseline, are helpful for statistical power to assess trends/trajectories.39 Longer PRO questionnaires have higher attrition rates for patient responses, so preference for shorter tools is prudent, particularly for repeated measures.38 For example, the FACT-G7 has been validated for internal consistency as a shorter version of FACT-G (which has 27 items).40 When multiple scales exist for similar domains, such as EORTC QLQ-C30 and FACT-G, comparison studies can often be24 found but ultimately length, prior use in a similar clinical setting, and appropriateness of precise questions included can guide one's PRO tool selection.

Due to PROs representing patient-centered outcomes, their use as secondary and exploratory endpoints on therapeutic clinical trials has increased exponentially in the last 2 decades. The FDA's Center for Devices and Radiological Health (CDRH) observed a >500% increase in the number of applications incorporating PRO data from 2009 to 2015.41 A classic example is ruxolitinib for myelofibrosis—although the primary endpoint was measured spleen size, the secondary endpoint of total symptom score was equally important for establishing a rationale for drug approval.42,43 PROs are also a potential solution to under-reporting of frequency and severity of adverse events by clinicians compared with patients,44,45 and have been shown to detect adverse events earlier in patients’ treatment course.44 Therefore, to enhance the accuracy and transparency of toxicity reporting in clinical trials, the National Cancer Institute developed a patient-reported version of the common terminology criteria for adverse events, or PRO-CTCAE.34 PRO-CTCAE consists of various items that prompt patients to rate specific symptoms based on severity, frequency, and the magnitude to which they interfere with their life.46 However, PRO-CTCAE has been noted to be optimally used as a complement, rather than replacement, for clinician-reported CTCAE.

Treatment toxicity represents an increasingly common patient-centered primary endpoint in radiation oncology trials, with examples including RTOG 0529 (which evaluated grade 2+ adverse events with IMRT) and NRG CC-001 (which aimed to improve neurocognition with hippocampal sparing). However, classic PROs are still not common primary endpoints of therapeutic clinical trials in either radiation or medical oncology. This is in contrast to clinical trials in palliative care, which often rely on PROs as the primary endpoint.47 Barriers to specifying a PRO as the primary endpoint of a trial include concerns about missing data,48 particularly for nonacute endpoints given evidence that patient response rates decline over time.44 Additionally, preliminary data may not be as readily available to provide effect size estimates for initial sample size and power calculations. Despite the increasing importance of PROs in phase II and III clinical trials, integration of PROs into phase I clinical trials during the early assessment of therapy safety and tolerability remains low.49 Early incorporation of PROs into pilot and early clinical trials may provide insight into dose tolerability and aid in developing phase II and III trial designs.

While oncology practices aim to carefully monitor for disease progression and treatment toxicities, scheduling frequent visits can be challenging given the demand for clinicians’ time, resulting in periodic routine follow-up. Initial studies challenging this paradigm tested PROs as remote monitoring devices that could prompt visits when needed, and garnered significant interest when 2 early studies demonstrated an overall survival benefit. In the first clinical trial at Memorial Sloan Kettering Cancer Center, 766 patients with cancer undergoing chemotherapy for advanced solid tumors were randomly assigned to either the intervention arm, in which they electronically reported 12 common symptoms experienced during chemotherapy, or the control arm, in which they received routine follow-up as the standard of care.50 Importantly, when a patient-reported symptom worsened by a predetermined threshold, an email alert was sent to nursing staff for further management. At 6-month follow-up, study results showed improved in HRQOL among patients in the intervention group compared to the control group. In addition, subjects in the intervention arm had fewer emergency department visits and hospitalizations and remained longer on scheduled chemotherapy regimens. But importantly, and perhaps unexpectedly, on long-term follow-up, median overall survival was improved by 5 months with weekly PRO monitoring compared to usual periodic follow-up visits.4 Shortly thereafter, a similarly randomized multicenter French study51 met its primary objective of improved overall survival at 2 years. Importantly, this study also found that patients’ performance status at first detected relapse was high (ECOG 0-1) for over twice as many patients in the PRO-based experimental arm compared to usual care (76% vs 33%, P < 0.001). The leading hypothesis for the mechanisms by which PROs may influence survival is early detection of symptoms prompting action with additional life-extending treatment and preventing further adverse outcomes and functional decline. An ongoing large multicenter pragmatic randomized trial called PRO-TECT is being conducted to confirm these survival results are generalizable, though has already reported secondary endpoints of improved physical function, symptom control, and HRQOL.52

Clinical trials have also investigated the impact of distributing PROs assessing more global HRQOL measures. A prospective RCT from the United Kingdom53 demonstrated that routine monitoring of HRQOL via PROs (EORTC QLQ-30, HADS) enhanced physician-patient communication without extending the length of visits and found a clinically significant improvement in HRQOL (measured by FACT-G). In this study, participants were explicitly assigned to 1 of 3 groups: (1) Intervention Arm, in which subjects complete the PRO questionnaire and feedback is sent to the physicians; (2) Attention-Control Arm, where subjects complete the PRO questionnaire, but the feedback was not sent to the physicians; and (3) Control Arm, in which neither occurs and instead, the standard of care is followed. A key finding from this study was increased HRQOL improvement in both the Intervention and Attention Control group compared to the standard of care arm. Importantly, the intervention led to more frequent discussions with physicians about treatment side effects (which did not prolong visit duration). And interestingly, physicians reported that PROs did not play a significant role in their clinical decision making (P = 0.60).

Several other randomized trials have now been reported testing PROs as a symptom assessment intervention54, with several highlighted in Table. Many, but not all, studies have shown positive results. More work appears needed to optimize PROs in hospitalized patients.55

Real-time symptom reporting and ease of use are important factors in ePRO acceptability. Studies56, 59, 60 have shown that more than 80% of study populations found PRO questionnaires easy to understand. However, one study found that black patients reported a lower understanding of ePRO questions than white patients (56% vs 92%) and more challenges in understanding the symptom summary report (38% vs 72%), which was thought to contribute to less ease of use by black patients compared with white patients (71% vs 92%).61 Other factors known to contribute to ePRO completion rates and understanding include education status, employment status, computer literacy and health/e-health literacy, which should be considered when designing and implementing ePRO tools in health care to prevent paradoxically worsening disparities with these digital tools. Potential solutions include providing ePRO information and ongoing technical assistance.

As highlighted in semi-structured interviews, PROs may improve patient-clinician communication by promoting structured thinking and helping to identify symptoms that physicians may have overlooked.61 PROs have the potential to significantly improve patient understanding of their disease, enhance one's sense of control over their care, and improve patient satisfaction.60,61 The majority of patients said they would use PROs again during follow-up visits and recommend them to others.60 Therefore, it is imperative to implement these efforts to maximize the benefits of ePRO tools in health care and ensure an equitable access for all patients.

Clinicians have reported that ePRO data, when used appropriately, can improve the quality of patient care and clinician efficiency.60 Interpreting a PRO report before a patient encounter might foster a more systematic thought process and a focused discussion of the important clinical issues that must be addressed during the visit.60,62 In addition, one study60 found that PRO information was helpful in reducing documentation burden.

While research has shown that incorporating PROs into clinical encounters does not significantly increase the length of patient visits,53 clinicians have reported challenges in addressing all PRO components within a single appointment due to time constraints.63 To streamline the integration of PROs into healthcare settings, recommendations suggest integrating PROs into the electronic health record (EHR) as a single platform for navigating and interpreting PRO data rather than relying on multiple devices.63 However, this implementation process has its own administrative system-level barriers, such as the lack of resources and technological support.63

Clinicians tend to seek user-friendly data displays, such as symptom trajectories and integrated red alerts to flag clinician attention to facilitate workflow efficiency.64,65 However, alerts can be excessive, with 26 of 55 nurses (47%) in one study60 reporting too many notifications, while only 19 (35%) reported an appropriate number of alerts. Nonetheless, 40 (71%) said alerts were beneficial in directing symptom management. Appropriate symptom thresholds are important, as is establishing workflows that direct the right data to clinician at the right time.

Although ePROs have gained widespread acceptance, practical implementation can be challenging. Previous studies have identified many barriers that can occur at any of 3 levels: the patient, the clinician, and the health system (Fig.).63,66, 67, 68, 69

To overcome patient-specific barriers, it is critical to address patients’ perception of PROs. Disclosure and emphasis on the fact that their privacy and relationship with their provider would not be jeopardized may improve the acceptability of ePROs use in routine cancer care.66 Educating stakeholders on the need for PRO collection by demonstrating the benefits of disease tracking and emphasizing how PROs can help to improve care delivery by preventing downstream adverse events is needed. To address digital literacy, conducting PRO usage training sessions can help to improve skills such as utilizing devices, logging in, and navigating the questionnaire. Patients may see PROs as time-consuming due to the nature of their disease and treatment regimen; it is therefore critical to be explicit about the domains to be assessed and the frequency of assessments. For example, limiting the use of generic PROs and encouraging weekly assessments (instead of daily, when appropriate) can help to alleviate time constraints and risk of survey fatigue.38 In addition, utilizing proxy reporters in older individuals may be acceptable (ie, either spouse or significant other or caregivers), specifically when proxy individuals are the main channels of communication for patients.3

Including participants from different ethnic groups in clinical trials is crucial to ensure the applicability of study results to a diverse population. However, a recent systematic review outlines several barriers to promoting inclusiveness in the use of ePROs,67 including (1) limited availability of translated PROs, (2) research staff training, and (3) financial constraints. Although some PRO tools have been validated in multiple languages (ie, EORTC and PRO-CTCAE), many are available only in English, and ad hoc exact translations may not capture the intended meaning of the questions in different populations.69 The Professional Society for Health Economics and Outcomes Research (ISPOR) Taskforce has outlined recommendations for the PRO translation and cultural adaptation process to ensure linguistic and cross-cultural validity when adopting translated PRO tools.69 Language barriers are often exacerbated by systemic barriers such as research and technical support needed to distribute and interpret responses, further highlighting the need for adequate resources to ensure equitable implementation.