Innovations in the Generation of High-Quality RWE

Real-world evidence (RWE) was one of the key trends at the 2022 Vienna International Society for Pharmacoeconomics and Outcomes Research (ISPOR) meeting. Sessions on RWE ranged from strategic perspectives on how RWE can be best utilised, as explained here (‘Collaboration Is Key – Real-World Evidence for Decision-Making’) to explorations of innovations in the generation of RWE. Innovations in the generation of RWE included templates for developing and reporting RWE, such as the target trial approach and HARmonized Protocol Template to Enhance Reproducibility (HARPER), machine learning (ML) approaches and new statistical methods for quantifying bias in indirect treatment comparisons. These innovations have the potential to address some of the issues that have been historically associated with RWE, such as data quality or risk of bias, and thereby accelerate acceptance of RWE and broaden use-cases. However, many sessions highlighted that RWE should continue to complement rather than replace evidence from randomised controlled trails (RCTs).

RWE is increasingly becoming a legitimised and formalised source of evidence for payers, regulators and Health Technology Assessment (HTA) bodies. The US Food and Drug Administration (FDA) for example recently approved Prograf (tacrolimus) in combination with other immunosuppressant drugs for preventing organ rejection in adult and paediatric patients receiving lung transplantation based on supporting evidence from a RWE study.¹ The results of the study were considered adequate and well-controlled and were considered alongside supporting data from RCTs investigating Prograf in other solid organ transplant cases. RWE is also a key aspect of the European Medicines Agency (EMA) clinical evidence vision for 2030, and the National Institute for Health and Care Excellence (NICE) in the UK recently published a RWE framework to help improve the quality of RWE informing their guidance.^2,3 Costello Medical recently published a dedicated commentary on the NICE RWE framework, which can be found here. There were many initiatives presented at ISPOR aiming to standardise and improve the quality of RWE so that its value can be fully.

Figure 1: Innovations in the generation of high-quality real-world evidence (RWE) presented at the 2022 Vienna ISPOR meeting

RWE Study Design and Reporting Templates

A number of RWE study design templates were explored at ISPOR, including the target trial approach and HARPER. The templates aim to improve the quality of RWE and reduce uncertainty, for example, when used to inform comparative efficacy assessments in HTAs. Historically, the preferred sources of evidence for these has been RCTs, as RWE studies are more susceptible to confounding or bias.⁴

The target trial approach is a framework for designing more robust comparative studies using real-world data (RWD).⁴ This involves designing the protocol for a hypothetical RCT and then emulating the target trial using the available RWD. Following a target trial approach can help reduce the risk of biases common in RWE studies such as confounding or selection bias by ensuring the research question and methodology are well-defined and can capture causal relationships, as explained here (‘Using Real-World Evidence in Health Technology Assessment and Regulatory Decisions’). The RCT DUPLICATE initiative ‘Randomized, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology’ for example uses this approach to compare the findings of RCTs relevant to regulatory decision-making with the findings of noninterventional RWE.⁵ The target trial approach was also recently used to demonstrate the transportability of overall survival estimates from US to Canadian patients with advanced non-small cell lung cancer.⁶

HARPER further provides a harmonised protocol template for RWE studies that evaluate a treatment effect.⁷ Following HARPER facilitates clear communication about data provenance (documentation of where data came from), design, analysis and implementation of RWE and therefore reproducibility, replication using independent data and assessment of potential sources of bias. A number of other templates were discussed, such as the SPACE framework for identifying fit-for-purpose data and the StartRWE template for planning and reporting RWE studies.^{8, 9} Collectively, these have the potential to improve the robustness and transparency of RWE, but it remains to be seen which of them will be widely adopted.

Machine Learning Methods

Another common theme at the meeting was how ML methods can be used to drive innovation in RWE. One application of ML in this context is to efficiently extract information from electronic health records (EHRs) and generate variables for analysis.¹⁰ EHRs often include unstructured data that contain data elements which are critical for outcomes research. Extracting this information currently involves trained experts manually labelling this information, which is costly and resource-intensive. ML algorithms can be trained on labelled datasets to learn the language patterns associated with a given variable and then applied at scale to new datasets. Other potential applications of ML that were presented at the meeting included using ML to automate systematic literature reviews or predict the risk of clinical outcomes such as stroke using large administrative health databases. Whilst many of these have the potential to enable RWE studies that previously would not have been feasible or possible, it remains to be seen whether concerns about data quality and interpretability of ML methods can be overcome for the wider adoption of ML-extracted data.

Single-Arm Trials and Indirect Treatment Comparisons

Lastly, multiple sessions reflected on the increasing trend of single-arm trials (SATs) informing treatment efficacy in oncology HTA submissions, and how RWE can be used to compare to existing treatments. In an SAT, a sample of individuals with the targeted medical condition is given the experimental therapy and followed over time, but there is no control arm.¹¹ SATs have some advantages over RCTs as they avoid allocating patients to potentially less effective control therapies and address difficulties with small sample sizes in rare disease or rare subsets of a disease. The FDA granted 153 new oncology indications between 2001–2020 based on SATs.¹²

The lack of a control arm is a key limitation of SATs, since these trials do not directly compare new treatments to existing treatments and preclude anchored indirect comparisons. Treatment effects must therefore be derived using data from a different source and rely on unanchored indirect treatment comparisons, which are highly susceptible to bias through confounding. Common methods to reduce the impact of potential confounding include matching-adjusted indirect comparisons (MAIC) or simulated treatment comparisons (STCs). These methods rely on the assumptions that all prognostic factors (PFs) and effect modifiers (EMs) are accounted and adjusted for. However, there remains risk of bias as without random allocation of patients to treatment groups, there may be other unobserved differences in factors that impact the observed treatment effect (unmeasured confounding).

In a session on innovative methods in indirect treatment comparisons, Kate Ren from the University of Sheffield presented research on how the analyses of SATs could be made more robust using an extended STC (ESTC) approach that allows for the formal quantification of the bias associated with unmeasured confounding in indirect treatment comparisons.¹³ This ESTC approach includes using both observed and unobserved PFs and EMs as covariates, simulating covariates for outcomes using Copula and G-estimation methods and running sensitivity analyses on the impact of unobserved PFs and EMs to obtain treatment effects. A re-analysis of the data from a Phase III RCT in metastatic colorectal cancer where the control arm was dropped and replaced with summary statistics from an external source showed that, when the ESTC approach was applied using this external source regardless of the values of the simulated covariates, the intervention was more efficacious than the comparator. Whilst more case studies are required to investigate the potential of this approach to accurately quantify bias, innovative approaches such as this one could help make single-arm trials more robust.

Conclusion

RWE is becoming an increasingly important source of evidence in health economics and outcomes research (HEOR) and a range of new innovations in the generation of RWE were presented at the ISPOR 2022 meeting in Vienna. These have the potential to drive improvement in RWE quality through guidance on generation and reporting of RWE, the application of ML capabilities to improve efficiency of RWE generation and new statistical methodologies to reduce uncertainty. However, it is also important to remember that whilst these innovations may drive acceptance and extend the use-cases for RWE, they cannot eliminate all uncertainties associated with the use of RWE. Therefore, RWE should not be seen as an alternative to an interventional study such as an RCT but should be used alongside RCT evidence to address data gaps within the evidence base. These innovations in RWE generation have yet to gain wide acceptance but indicate a positive trend in the quality of RWE generation. Future ISPOR meetings will most likely see more presentations on innovative approaches for better quality RWE.