Incorporation of Minimally Important Differences in Network Meta-Analysis

This article discusses the use of minimally important differences (MIDs) in network meta-analysis (NMA) for health technology assessment (HTA). This summary provides a useful entry point to our original research published in the BMC Medical Research Methodology, for incorporating MIDs in commonly available NMA ranking metrics.

What are minimally important differences (MIDs) and how are they used in NMAs?

MIDs represent the smallest value in a given outcome that is considered by patients or clinicians to represent a meaningful difference between treatments. Although using MIDs to specify hypotheses of superiority or non-inferiority is common in clinical trials, no such precedence exists for NMAs. NMAs typically only assess whether there are statistically significant differences between treatments, and do not generally consider minimally important differences between treatments.

Figure 1: An illustration of minimally important differences in a clinical context, using HbA1c as an example.

Can MIDs be used when comparing treatments in NMAs?

Yes, a conceptual framework, defined by Uhlman et al., for such an approach within NMAs has previously been published.¹

Is use of MIDs common in NMAs for HTA?

No. To our knowledge, very few NMAs conducted as part of HTAs incorporate MIDs to aid interpretability of NMA results.

MIDs are referenced in IQWIG HTA guidance documents, but not in the context of NMAs. However, more recently, as mentioned in our recent article relating to methods guidance for JCA, the JCA guidance suggests that in population-adjusted analyses, or if there is a risk of bias or confounding in a network, specifying a threshold away from the typical line of no treatment effect (i.e., an MID) is recommended.

Additionally, GRADE (grading of recommendations assessment, development and evaluation), an approach to draw conclusions from NMAs, highlight the relevance of contextualising the effect size (trivial to no effect, small but important effect, moderate and large effect) through appropriate specification of clinically meaningful thresholds.²

Despite the limited precedence of MIDs in NMAs for HTA, we consider there to be value in such an approach for researchers and decision makers. Given that underlying trials informing an NMA are non-inferiority, or superiority trials, why do we not set similar hypotheses when comparing treatments via NMA? Should the same set up of non-inferiority or superiority not also be considered? An argument against this would be that for NMAs we’re often only interested in the resulting point estimates and confidence intervals to inform cost-effectiveness models. However, from a narrative perspective – whether a treatment is superior, non-inferior or merely ‘statistically significantly different’ from its competitors is important, and so we would argue in favour of defining similar hypotheses as per clinical trials, when conducting NMAs. That is, by defining an MID within the hypothesis for a comparison verses a competitor treatment, the narrative and its relative positioning versus a competitor is enriched. Beyond the narrative perspective, being able to classify relative relationships between treatments in this manner could also help to make it clearer when HTA approaches that rely on an assumption of comparable efficacy (such as the use of a cost minimisation/cost comparison approach to health economic evaluation) are appropriate.

HTA bodies have only to gain if pharmaceutical companies consider MIDs when comparing their treatment versus competitors and submitting their evidence package. There may be challenges (e.g. subjectivity in defining an MID), but doing so limits the risk of interpreting meaningfulness of differences on the basis of statistical differences between treatments when none of clinical relevance exists. As such, HTA authorities would have greater clinical insight on the comparative value of a treatment, and greater clinical understanding of its relative positioning amongst other treatment options.

Furthermore, from a strategic perspective for manufacturers submitting to HTA bodies, if they are able to demonstrate that their product is not only statistically superior, but also clinically superior, then that only helps to strengthen their case for the benefit profile of their treatment. Similarly, where a treatment may be less strong on clinical outcomes than its competitors, manufacturers may be able to show that despite statistically unfavourable results a clinical conclusion of non-inferiority is appropriate. Inclusion of MIDs can provide clarity on the clinical positioning of a treatment, which when relying only on statistical differences (as per current convention) is missing from evidence packages submitted to HTA authorities.

What are ranking metrics in NMAs?

Outputs of NMA include estimates of relative treatment effects such as mean differences, and odds, risk or hazard ratios, which are provided to quantify relative differences in the performance of multiple treatments. Along with estimates of the relative treatment effects, ranking metrics are often provided to support or illuminate the relationships between treatments in terms of particular outcomes. Common ranking metrics for a given outcome include the probability of a treatment being best (among the assessed treatments), the probability of being ranked in at least a given position, or the Surface Under the Cumulative RAnking curve (SUCRA) values, which represent the proportion of competing treatments that a treatment outperforms (see Figure 2 for an illustrative example).

Figure 2: SUCRA curves for four generic treatments, illustrating the proportion of competing treatments that a treatment outperforms.

Why are ranking metrics important?

Network meta-analysis and the interpretation of its results influence (along with myriad other factors such as cost, clinician experience etc.) perceptions of the relative effectiveness and safety of treatments in an indication. How treatments are perceived contribute to their positioning among existing treatment options and therefore accessibility to patients, propensity of usage by physicians, and associated market share for pharmaceuticals.

Can MIDs be accounted for when generating ranking metrics?

Some literature exists to date in doing so.^3-6 In collaboration with colleagues from the University of Freiburg and the University of Waterloo, we have developed an R package and framework for incorporation of MIDs when generating associated ranking metrics from NMAs. The basis for this research was that by convention MIDs are typically ignored when generating ranking metrics (such as probability best, SUCRAs etc). Our article which has just been published in the BMC Medical research Methodology, overcomes this limitation.⁷

Short summary of our research article: Ranking of treatments in network meta-analysis: incorporating minimally important differences

Within the paper we provide a conceptual framework for accounting for MIDs when generating common ranking metrics in NMAs. We then showcase examples of the framework and implementation within real-world applications, including in both diabetes and Parkinson’s disease. We also provide software implementation as an R package for ease of use, as an add-on to the multinma R package.

We finally conclude that integrating MIDs into NMA ranking metrics as an adjunct to existing approaches and methodology represents an advancement in the clinical interpretability of NMA results. Essentially, considering clinically relevant differences in ranking metrics allows for the creation of useful and interpretable treatment rankings to enrich existing methodologies.