There is a need to take into account characteristics of the study design to choose DOIs and HOIs to optimise the performance of drug safety signal detection studies. This can be done through the application of a taxonomy framework. The aim of this work was to describe such a framework, to outline the main general principles considered and to apply these for the selection of a reference set in a self-controlled signal detection study. This work was based on previous studies but tailored to our specific method-based approach and to the specificities of signal detection. The general principles can be applied to any method but specific implementation needs to be adapted depending on the chosen design.

Previous work from the Sentinel project, despite not being directly related to drug safety signal detection, provides a good basis. We used Sentinel principles as the foundation for our framework, by reviewing their criteria and including only those relevant to signal detection. We added an assessment of the suitability of the DOIs and HOIs in the database of interest, as it is necessary to make sure the study is feasible or to choose a database accordingly.

While there is a clear need to change the way reference sets are developed for a method evaluation based on the exposure and outcomes of interest as well as the data source, these must be clearly defined a priori. Explanation of the choice of reference set should be consistent and transparent, in line with the renewed focus on enhanced replicability and transparency in real-world evidence generation from healthcare databases [22].

Several criteria from the Sentinel project were reused either directly or indirectly regarding the study requirements: the exposure use pattern, the onset and duration of exposure risk window, and the degree of misclassification through the coverage in the database of interest. We did not consider the strength of within- and between-person confounding because in our study time-invariant confounding is handled by the self-controlled design, and part of the time-variant confounding is handled by the short observation period, and the use of active comparators. If between-person comparative methods are chosen, these would be important to be considered when selecting the DOIs and HOIs. By applying this taxonomy framework, we are able to utilise a single study design to investigate a very broad list of preselected HOIs, covering a range of organ classes and different levels of seriousness.

The Mini-Sentinel work recommended that when assumptions are met, self-controlled designs should be used in priority because within-person confounding is handled [23]. If assumptions are not met, an alternative design should be chosen. Our work provided an example of checking SCCS assumptions and an extension to accommodate active comparators.

Although we attempted to present the choice of criteria in a systematic approach, some criteria still relied on clinical opinion and human judgement to be implemented, introducing some degree of subjectivity in the process, which was highlighted when applying the framework to the case study. Importantly, our proposed framework encourages the systematic documentation of the decision-making process, which could lead to high transparency of signal detection-related research studies.

In the application of the framework for the reference set, the original number of adverse events in the label investigated was 273, but the final list contains only 28 outcomes after application of the framework. However, it must be recognised that not all adverse events listed in a product label are of equal public health importance, and that many outcomes were excluded from our reference set for reasons such as their mild self-limiting nature. Here, we have selected those that the assessors considered to be of highest public health importance, as well as those that are most suited to the constraints of the study. The reference set we obtained contains HOIs and DOIs appropriately covered in the CPRD. We have also checked that the final list of drugs and outcomes was also well covered in the other databases (Systeme National des Donnees de Sante and MarketScan) used in the study. This will enable the use of the same reference set in several nationwide databases, leading to multi-database comparisons of the results of this study.

There is a potential misclassification and imbalance between the number of positive and negative controls [24] in the reference set obtained. Indeed, labels do not necessarily represent true causal associations, and an outcome absent from the label could still be associated with a DOI if the association is unknown. We identified a much larger number of positive controls than negative controls, which was because of the strict criteria we chose to ensure the quality of the negative controls. More broadly, we recognise that routine signal detection within a RWD database should be considered one of many tools in the broader signal detection armoury, with unique strengths and limitations.

This framework can be implemented at a broader scale in signal detection studies. Our work was method based (SCCS with active comparators as an example) but it is also possible to adapt a similar framework to drug- or outcome-based approaches, as well as to study designs other than SCCS. Depending on the drug(s) and/or outcome(s) of interest, their characteristics and coverage in the available database, one can choose the appropriate design and type of analysis. To apply our framework, a list of pre-specified outcomes of interest is needed. We selected the outcomes of interest based on drug labels, which is not necessarily useful when designing a non-performance assessment signal detection study. Further, there is no reason why this framework could not be applied to vaccine signal detection as well. Where a range of different drugs and outcomes are of interest that do not share characteristics amenable to a single study design, several study designs may need to be considered to optimise the potential for signal detection using routinely collected health data.