To the authors’ knowledge, this is the first study to comprehensively assess the acceptance of ITC methods in oncology by HTA agencies across England, France, Germany, Italy, and Spain. These countries were selected as the principal pharmaceutical markets in Europe and because of the availability of HTA evaluation reports published by their HTA agencies. A systematic approach was utilized for the search, review and selection process, extraction of data, and analyses. The results were reported quantitatively and qualitatively as robustly as possible based on the available data.

The analysis showed that an ITC was presented in 22% of the identified HTA evaluation reports assessing treatments for solid tumors in oncology. This suggests that direct comparisons were not sufficient and/or available in at least one in five HTA evaluations, and that an ITC was therefore performed to compare the efficacy of the intervention against the appropriate comparator(s). As such, although direct comparison remains the gold standard approach, the use of ITC methods is not uncommon, consistent with HTA methodological guidance for ITCs published over the last ten years [1, 2, 12, 21].

An exploratory analysis was conducted to investigate how manufacturers may submit different ITCs to different HTA agencies. For this analysis, the case of pembrolizumab in combination with pemetrexed and platinum chemotherapy in first-line treatment of metastatic non-small cell lung cancer was selected; this was the only case where an ITC was submitted to at least four countries (England, Germany, Spain, and France; no data publicly available for Italy), as reported by HTA evaluation reports. The fact that only one case was identified as presenting an ITC in HTA evaluation reports for up to four countries may illustrate manufacturers being influenced by the differing receptivity of ITCs anticipated for different HTA agencies.

For England, an NMA was initially presented, comparing pembrolizumab in combination with pemetrexed and platinum chemotherapy versus a number of different chemotherapies, with 17 studies included within the network. A second ITC was presented using the Bucher method, comparing pembrolizumab in combination with pemetrexed and platinum chemotherapy versus pembrolizumab monotherapy. A Bucher ITC for the same comparison of pembrolizumab in combination with pemetrexed and platinum chemotherapy versus pembrolizumab monotherapy was also presented in France, Spain, and Germany. For England, the ITC included KEYNOTE-189 and KEYNOTE-024 trials [29, 30]. Contrastingly, the evaluation report for Germany also included KEYNOTE-021G and KEYNOTE-042 [31, 32]. NICE criticized the absence of KEYNOTE-021G, stating that these data should have been included within the ITC analysis. No such details were presented in the French and Spanish HTA evaluation reports. This highlights that the decision made by manufacturers as to whether to present an ITC, may depend on the treatment landscape and relevant comparators within the relevant country, the acceptability of ITCs by HTA agencies, data availability, and the acceptability of the inclusion and exclusion criteria for different sources of data (e.g., clinical trials). Further analysis of additional cases may extend upon the reasons provided here; however, in-depth analysis remains limited due to the lack of data provided in many HTA evaluation reports.

Differences arose between countries in the proportion of HTA evaluation reports that included an ITC, with the lowest number of HTA evaluation reports with an ITC being in France (6%). The results from this analysis are influenced by whether or not manufacturers decide to submit an ITC to an HTA agency, in anticipation of the receptivity to such analysis. The results by country may vary because of differences in the submission specifications (such as the most appropriate comparator in clinical practice in each country), but also because of differences in the evaluation conducted by the HTA agency. Fewer ITCs were submitted in French reimbursement dossiers, despite the Doctrine of the TC and HAS methodological guidance for ITCs stating that “the use of indirect comparisons may be considered” [8, 12]. In Germany, 18% of HTA evaluation reports included an ITC, which is unsurprising given the IQWIG general methods guidelines state that “ there are still several unsolved methodological problems, so that currently the routine application of [indirect comparison methods] within the framework of benefit assessments is not advisable” [1]. Conversely, the highest number of ITCs were identified in England (51%). This reflects NICE recommendations which explicitly state that “data from trials that compare the technology with non-relevant comparators may be needed to enable the technology and the comparators to be linked in an indirect or mixed treatment comparison.” NICE also recommends that full details of the ITC methodology should be presented in the manufacturer HTA dossier [2].

Among the five countries of interest, the overall acceptance rate of the included ITCs was estimated to be 30%. This indicates that although head-to-head comparisons are preferred, the use of an ITC can be considered to inform HTA recommendations. Nevertheless, while ITC methods are evaluated by HTA agencies, their acceptance rate remains suboptimal. The highest acceptance rate was observed in England (47%) and the lowest in France (0%). Although HAS has published methodological guidance for ITC methods to date, the acceptance of ITC methods in France remains poor. Whilst a direct link cannot be made from the evidence collected in this analysis, this low acceptance of ITC methods by HAS raises further questions into the overall reimbursement of new oncology interventions in France, which may justify further investigation.

The analysis of ITC techniques utilized showed that the most commonly used ITC techniques across the five countries were NMA (23%), Bucher ITC (19%), and unadjusted ITC (14%). In particular, Bucher ITCs were the most commonly used technique in Germany (57% of ITCs), which is in line with IQWIG guidelines for ITC methods. NMAs were reported in 45% of reports identified for France; however, these results should be interpreted with caution due to the comparatively low number of HTA evaluation reports analyzed (N = 11). Research on the methodology of ITC techniques indicates that these are evolving quickly and that improved techniques may be available in the future, particularly in the area of multi-level network meta-analysis [33,34,35]. Moreover, further work is also needed to provide a detailed comparative analysis of the practical performances of different ITC techniques and to reduce the substantial risk of bias associated with ITCs conducted using nonrandomized data [36, 37].

The analysis of HTA agency criticisms of the ITC methods used showed that the most common criticisms across the five countries related to heterogeneity/risk of bias, lack of/unclear data, and the statistical methods used. This finding was consistent with Werner et al., which also found that study similarity and statistical methods were amongst the most common reasons for rejection of ITCs by IQWIG [38]. This may be due to the inappropriate selection of statistical methods and the failure to address heterogeneity via the chosen methodology. Alternatively, statistical methods may be subject to criticism in cases where a more appropriate method is unfeasible, for example due to a paucity of data. Although various statistical methods may be associated with minor sources of bias, they are generally suitable when used within the appropriate sphere of application. As such, there appears to be a need for further clarity on the appropriate use of statistical methods when conducting ITCs and for guidelines on the assessment of the results of such methods.

Across this analysis, the overall acceptance rate of ITCs was generally low, and a range of criticisms from HTA agencies were identified. This suggests that, whilst in the absence of a direct comparison ITCs may bring additional relevant evidence, there remains a need to select appropriate ITC techniques, applied to good quality data, with adequate considerations regarding heterogeneity and risk of bias. Since most HTA evaluations are based on ITCs submitted by the manufacturer, which are not necessarily published in peer-reviewed journals, this may increase the risk of the inappropriate use of ITC methods. There is also a need for an international consensus on the appropriate use of ITC methods, which could improve the quality of ITCs submitted to HTA agencies and the rate of ITC methods acceptance. Research to compare the performance of the different statistical methods is needed to establish the preferred method(s). This analysis found higher acceptance rates of adjusted ITCs (including Bucher ITCs), compared with unadjusted ITCs, in England, France, and Germany, which is in line with currently available methodological guidance. This was consistent with Lebioda et al., which also found adjusted ITCs to be more generally accepted than unadjusted ITCs [39]. Although justification for the choice of ITC methodology is rarely provided in HTA evaluations, it is likely that this depends on the availability of data and limitations in the network.

The main source of clinical evidence used in the included ITCs was RCTs (52%). Surprisingly, a large proportion (32%) of HTA evaluation reports did not describe the source of clinical evidence (e.g., whether the clinical data were derived from an RCT or single-arm trial). Seventeen percent of the HTA evaluation reports used data from single-arm studies to estimate the relative treatment effect in the ITCs conducted. This is consistent with the growing number of single-arm studies conducted in oncology and the growing number of oncology drug candidates that are obtaining regulatory approval based on noncomparative trials. In terms of the ITC techniques found to be accepted in cases of single-arm studies, the most commonly reported technique was a MAIC (included in 13% of the HTA evaluation reports). STCs, another indirect comparison approach typically utilized with single-arm studies, remained largely underrepresented (included in only 3% of the HTA evaluation reports). For the ITCs that were based on single-arm studies, data for the external comparator arm were more often derived from a clinical trial than from a RWE study, which reinforces findings from Patel et al., which found that 24% of submissions in oncology (including hematological cancers) used a clinical trial as the source of external comparator data, versus 20% that utilized RWE data [40].

It should be noted that new guidelines for the use of ITCs have recently been developed by the EUnetHTA 21 to be applied from January 2025 with the implementation of the EU HTA regulations [11, 41]. It would be pertinent to reassess the findings of this study after the new regulations have been implemented for new oncology products and advanced therapeutic medicinal products.

There are several limitations to this analysis. Firstly, a limited number (<10) of HTA evaluation reports were identified for a number of ITC techniques, including NMR, STC, IPTW, and adjusted ITCs (method not reported). Moreover, one category of ITC techniques was “not determined” due to no further details being available in 30% of the HTA evaluation reports, most notably within Italian HTA evaluations. This category was kept in the analysis to avoid any bias in data interpretation. This lack of information illustrates the need for more transparency in published HTA evaluation reports, although it may also reflect the wide use of unpublished evidence from industry in HTA decision making. Additionally, it should be noted that the proportions of reported ITC techniques may be reflecting the available evidence structures for the HTA evaluations, rather than the particular strengths or weaknesses of the techniques themselves. Secondly, many HTA evaluation reports only stated that an adjusted ITC was used, without additional information on the method used, which limits the assessment of the value of the approach used and again, calls for more transparency from HTA agencies in terms of the information included in their publicly available HTA evaluation reports, as well as their guidance for the reporting of ITC methods. Thirdly, there was a paucity of data publicly available in the Italian HTA evaluation reports, with no information to determine the ITC techniques used in 93% (27/29) of the ITCs presented. This suggests a specific lack of transparency in the Italian HTA process, compared with the other countries in this analysis. Nevertheless, the HTA evaluation reports from Italy were still kept within the analysis, due to the lack of data within the literature that has assessed the use of ITCs included in Italian HTA evaluation reports. Additionally, the categories of criticisms were limited, due to unclear reporting across HTA agencies resulting in overlap between the categories provided. As such, categorization could be improved with additional transparency and evaluation of HTA reports over time. Nevertheless the analysis concludes that at least 40% of ITCs were criticized related to heterogeneity/risk of bias, lack of/unclear data, and the statistical methods used.

Finally, the analysis was limited to publicly available HTA evaluation reports, which may be biased compared with the total number of files submitted by the manufacturer. Also, the analysis was restricted to the oncology area specifically. Similar trends may be relevant to other therapeutic areas; however, this requires further exploration. Overall, this analysis was therefore limited by discrepancies in the ITC information available in the published HTA evaluation reports among the five countries of interest. Finally, the acceptance of the ITC methods was not clearly stated in most of the HTA evaluation reports, and it was not always reported whether the conclusions of the benefit assessment were partly inferred from the ITC, illustrating a lack of transparency by HTA agencies in terms of the weight of the ITC in the reimbursement decision making process among the full body of evidence. Greater transparency is needed to strengthen the evaluation of ITC methods and their acceptance by HTA agencies. There is scope for further evaluation assessing the correlation between the submission of direct versus indirect comparative evidence to HTA agencies and reaching positive recommendation for reimbursement.

At a regulatory level, while RCTs remain gold standard, the European Medicines Agency (EMA) has not yet adopted a formal position on the acceptability of ITCs [42]. However, for ITCs based on single-arm studies and RWE as an external comparator, RWE can have a substantial impact on regulatory decision making, by providing evidence on the natural history of disease and standards of care, and by contextualizing results of uncontrolled trials [42,43,44,45]. A study supported by the EMA examining marketing authorization applications (MAA) submitted in 2018 and 2019 showed that 40% of initial MAAs, and 18% of applications for an indication extension for products currently on the market, contained RWE, mostly sourced from registry-based cohort studies. RWE was used to support the efficacy of the relevant intervention in 50% of cases, for example, through provision of external comparators for single-arm studies [46]. Antineoplastic and immunomodulating agents accounted for over a third of initial MAAs and a half of indication extension submissions, and as such, many applications where RWE was used [46]. Many of these products are indicated for rare and often fatal diseases, with applications being based on uncontrolled clinical trials instead of traditional RCTs. Where RCTs are not available, RWE has the potential to help address uncertainty, for example, by providing supplementary clinical evidence for external comparators to single-arm studies. Depending on the role of the external comparator, the contribution could be designated as a “supportive” study or as part of the “main” study. Work is ongoing to detail the type, purpose, and influence of RWE during regulatory assessment for marketing authorization, but with the anticipated increase in RWE studies, it will be interesting to see how ITC methods may need to evolve to incorporate more RWE.