AI in HEOR: Pathways, Challenges and Future Directions

Early Studies into the Use of Artificial Intelligence in HEOR

The conference emphasised the growing trend of adopting artificial intelligence (AI) and machine learning (ML) tools, like GPT-4, across various Health Economics and Outcomes Research (HEOR) activities, such as systematic literature reviews (SLRs), economic model development, and data analysis. It was evident that these tools are gaining traction; in an early session at the conference, an audience poll found approximately 55% felt that generative AI is a useful tool and about 35% believed it would completely transform HEOR.1

Various groups reported having experimented with AI tools, with promising results.1, 2 The availability of AI/ML tools for systematic reviews is increasing. These have been applied at each stage of the review (search string generation, article screening, risk of bias assessment, and data extraction), with the use of AI/ML at the abstract screening step being most common.3-5 Additionally, case studies were presented on using GPT-4 to:

  • Generate code that could replicate incremental cost-effectiveness ratios (ICERs) from a published partitioned survival model within 1%6
  • Successfully automate stages of a network meta-analysis, including data extraction, generating an R script, executing the NMA and writing a short report of the findings7

Nonetheless, multiple studies found lower accuracy in AI-generated outputs compared to human outputs,8, 9 emphasising the need for careful prompt engineering to successfully use generative AI and the importance of maintaining human review of outputs.

NICE’s Engagement with Artificial Intelligence

The National Institute for Health and Care Excellence (NICE) provided various updates on its exploration into the impact of AI and machine learning (ML) technologies for health technology assessment, including work being done through the Next Generation Health Technology Assessment (HTx) project.10 NICE has been exploring the use of AI and ML in three main categories:11

 

1.

Improving internal processes

Internally, NICE has used AI and ML to automate various processes like deriving search strategies, randomised controlled trial (RCT) classification, and guidance surveillance. For example, they’ve piloted an algorithm to match recommendations from NICE and the Office for National Statistics to accelerate guidance surveillance for breast cancer screening.

2.

Evaluating the use of AI in technology appraisal submissions

In relation to company submissions, NICE is contemplating guidance updates to identify areas of high and low risk for AI use. While the use of AI and ML in submissions to NICE is currently limited, it is anticipated to increase significantly in the future.

3.

Assessing technologies with an AI or machine learning component

Lastly, NICE is reviewing how health technologies incorporating AI or ML components should be appraised. Updates to the CHEERS checklist are also expected in the near future, to better suit such technologies.

Embracing AI in HEOR – Navigating Challenges and the Path Ahead

While the potential of AI to transform the HEOR field is exciting, numerous challenges were also raised over the course of the conference. Among these are concerns around the accuracy and reliability of output from generative AI tools, reproducibility and transparency, and acceptance of AI outputs by health technology assessment agencies. A recurring theme at the conference was the need for retaining human review checkpoints throughout any process using AI to address some of these challenges, and there exists a need for upskilling health economists and related professionals in AI methodologies to leverage AI’s full potential and to foster trust in its outputs.

IT security and data confidentiality is another concern, particularly in relation to proprietary data inputs. Mitigation suggestions offered during the conference include avoiding AI/ML model training with publicy available models, using ring-fenced third-party models, and establishing a consistent company-wide policy on the use of AI tools. There was notably little coverage of the legal and copyright considerations around the use of AI, despite their importance, particularly in the context of processing published articles or other copyright material.

Despite these challenges, there was optimism that AI has the potential to disrupt the HEOR field in a positive way. Ongoing work is needed to identify exactly where AI provides a consistent improvement over conventional approaches and to ensure AI is implemented in a way that does not compromise on quality.

For more on the ISPOR learnings on the use of generative AI for health economic modelling specifically, please see our commentary “The Promise and Pitfalls of Generative AI for Health Economics”.

References

  1. Breakout Session 102. Embracing the AI Revolution: Exploring the Promising Future of Generative AI in HEOR. ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  2. ISPOR Forum 139. Artificial Intelligence-Enabled “Research Assistants” in Health Economics and Outcomes Research – Hire or Fire?. ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  3. Poster Presentation MSR46. Breaking Through Limitations: Enhanced Systematic Literature Reviews with Large Language Models. ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  4. Poster Presentation MSR80. AI-Enabled Risk of Bias Assessment of RCTs in Systematic Reviews: A Case Study. ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  5. Presentation P23. Evolving use of Artificial Intelligence and Machine Learning in Systematic Literature Reviews (SLRs). ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  6. Presentation P1. Automating Economic Modelling: A Case Study of AI’s Potential With Large Language Models. ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  7. Presentation P21. A Comparative Analysis of Large Language Models (LLM) Utilised in Systematic Literature Review. ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  8. Presentation P23. Evolving use of Artificial Intelligence and Machine Learning in Systematic Literature Reviews (SLRs). ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  9. Presentation P40. Man Versus Machine: Can AI-Assisted Technology be Used to Support the Development of Economic Models?. ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  10. Breakout Session 124. Artificial Intelligence to Support HTA and Conducting HTA for Artificial Intelligence Technologies: Recent Developments and Reflections. ISPOR Europe Congress, Copenhagen, Denmark, 2023.
  11. Breakout Session 206. The Place of Artificial Intelligence in HTA and HEOR. ISPOR Europe Congress, Copenhagen, Denmark, 2023.

If you would like any further information on the themes presented above, please do not hesitate to contact , Head of Technical Innovation and Development or Hannah Luedke, Consultant (LinkedIn). Sara Steeves and Hannah Luedke are employees at Costello Medical. The views/opinions expressed are their own and do not necessarily reflect those of Costello Medical’s clients/affiliated partners.

Loading