The Ongoing Challenges of Immature Data

The increasing use of flexible and accelerated regulatory pathways has prompted a reliance on immature survival data in HTA submissions. For a number of years now, ISPOR conferences have discussed approaches to handling survival data immaturity, including utilising surrogate endpoints or leveraging expert opinion. In Copenhagen this year, experts continued to explore the challenges arising from the use of these methods.

It is well established that the use of surrogate endpoints should be underscored by validation of the relationship between the surrogate outcome and final clinical outcome, based on biological plausibility, the precision of the surrogate measurement and level of correlation (Figure 1).1 Currently, validated surrogate endpoints only exist in a limited number of indications, including some oncology and cardiovascular indications. However, even validated surrogates have been found to over- or under-predict treatment effects when compared to data generated post-approval. In Session 126,2 Dan Ollendorf presented a specific case study of elevated low-density lipoprotein (LDL), an established surrogate outcome for cardiovascular risk.3, 4 Early modelling of treatments that are effective at significantly lowering LDL predicted significant reductions in most types of cardiovascular events based on the surrogate relationship.5 However, later modelling exploring the relationship based on recent randomised control trials has suggested that the relative risk of reduction in cardiovascular events was overpredicted, highlighting the need for continued research into these approaches.6 It will be interesting to see findings from similar case studies in oncology indications where post-approval data are now available.  

Figure 1. The Surrogate-Final Outcome Relationship

Diagram showing the Surrogate-Final Outcome Relationship 

Reference: ISPOR Value & Outcomes Spotlight, November 2020 

Importantly, it was noted that further guidance on best practice use of surrogate outcomes in economic analyses is currently being developed by a group of several international organisations and HTA bodies, including NICE, SMC, CADTH, HTAi Global Policy Forum, ICER and others. This work has been ongoing since January 2023, and we hope to see some output by next year.7 Of note, there were calls from representatives from HTA bodies for the pharmaceutical industry to deepen the exploration of clinical trial datasets in order to discover robust correlations between measured outcomes and maximise the potential of collected data rather than solely relying on existing surrogate endpoints.

Another topic of discussion at the conference was the appropriateness of methods for incorporating external data to produce plausible lifetime extrapolations. HTA bodies such as NICE have developed methods guidance on ways to improve extrapolations by incorporating external data, such as real-world evidence (RWE) and expert opinion, but there is a need for more in-depth, standardised guidance on the application of methods, i.e., when these methods are most appropriate and how to apply them in practice.8 External data approaches include pooling RWE with clinical trial data; using RWE to inform baseline risk; and incorporating general population mortality in extrapolations. Which existing methods enhance the clinical plausibility of extrapolations without diminishing structural uncertainty was a matter of considerable debate in Session 229,9 with some experts highlighting the need for more complex methods while HTA decision-makers leaned towards a “simpler” approach. A few issues were discussed in detail:

  • The ability of external data to improve generalisability: Some experts highlighted that complex methods such as left or right censoring techniques may be required to create a RWE dataset more generalisable to current practice; this is because while mature RWE is more informative, it is also often historic and this can raise generalisability issues. Decision-makers, however, indicated that it may be sufficient to incorporate external data to establish baseline risk and then apply relative effects through network meta-analyses10 
  • Complex methods for the pooling of RWE and trial data, such as Bayesian dynamic borrowing,11 were suggested as appropriate methods to account for heterogeneity between the source data and target study population. However, complex methods are limited in their ability to compensate for fundamental data limitations. The decision-maker perspective emphasised that maintaining clinical plausibility regarding absolute and relative survival effects is important above all, whichever methods are selected 

The sessions at the conference tackling the challenge of immature data highlighted the trade-off between complexity and simplicity: there is at times a misalignment between the academic discourse and the need for transparency from HTA agencies (who noted that more complex methods are frequently accompanied by insufficient and/or unclear explanation). Nevertheless, there was a consensus that manufacturers should participate in early HTA consultations to verify the use of methods in the case of immature data and to establish a clinically plausible base case. With the growing influx of innovative therapies, it is vital that decision-making bodies and manufacturers are able to work together to understand which methods are likely to be accepted and the level of evidence required, and further, clear guidance from decision-making bodies would be welcome in this regard.


  1. Ciani O, Buyse M, Drummond M, et al. Time to Review the Role of Surrogate End Points in Health Policy: State of the Art and the Way Forward. Value in Health 2017;20:487-495.
  2. Breakout Session 126. How Should Decision Makers Deal with Uncertainty in the Evidence to Support Surrogacy Relationships in Cost-Effectiveness Analysis. ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  3. Mortensen MB, Nordestgaard BG. Elevated LDL cholesterol and increased risk of myocardial infarction and atherosclerotic cardiovascular disease in individuals aged 70-100 years: a contemporary primary prevention cohort. Lancet 2020;396:1644-1652.
  4. Baigent C, Blackwell L, Emberson J, et al. Efficacy and safety of more intensive lowering of LDL cholesterol: a meta-analysis of data from 170,000 participants in 26 randomised trials. Lancet 2010;376:1670-81.
  5. Institute for Clinical and Economic Review. PCSK9 Inhibitors for Treatment of High Cholesterol: Effectiveness, Value, and Value-Based Price Benchmarks. Available here. Last accessed: 27 November 2023.
  6. Institute for Clinical and Economic Review. Evolocumab for Treatment of High Cholesterol: Effectiveness and Value. New Evidence Update. Available here. Last accessed: 27 November 2023.
  7. National Institute for Health and Care Excellence. 2022. News: How ‘surrogate outcomes’ influence long-term health outcomes. Available here. Last accessed: 27 November 2023.
  8. Rutherford, MJ., Lambert, PC., Sweeting, MJ., Pennington, R., Crowther, MJ., Abrams, KR., Latimer, NR. DSU Technical Support Document 21. Flexible Methods for Survival Analysis. 2020. Available here. Last accessed: 27 November 2023.
  9. Breakout Session 229. Incorporating External Evidence Into Extrapolations: A Debate on the Trade-Offs Between Complexity and Simplicity. ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  10. National Institute for Health and Care Excellence. TA930: Lutetium-177 vipivotide tetraxetan for treating PSMA-positive hormone-relapsed metastatic prostate cancer after 2 or more treatments. Available here. Last accessed: 27 November 2023.
  11. Chaudhary MA, Edmondson-Jones M, Baio G, et al. Use of Advanced Flexible Modeling Approaches for Survival Extrapolation from Early Follow-up Data in two Nivolumab Trials in Advanced NSCLC with Extended Follow-up. Medical Decision Making 2023;43:91-109.

If you would like any further information on the themes presented above, please do not hesitate to contact Sukanya Subramaniyan, Health Economist (LinkedIn) or Natalie Hearmon, Global Head of Health Economics and Statistics. Sukanya Subramaniyan and Natalie Hearmon are employees at Costello Medical. The views/opinions expressed are their own and do not necessarily reflect those of Costello Medical’s clients/affiliated partners.