SRs are a recognised research approach for identifying, synthesising and analysing published evidence on topics of interest.4 Their aim is to provide a comprehensive, unbiased and trustworthy assessment of available evidence to support decision-makers in adopting policies that ‘do more good than harm’.5 Indeed, policy makers, researchers, practitioners and public stakeholders from within and outside of healthcare, place increased emphasis on the importance of conducting SRs.4 Although the rationale for conducting SRs remains the same, the process of systematic reviewing has changed significantly in recent years.6 Technological advances and automation have helped to speed up processes such as study identification.2 However, additional steps in the review process, reflecting new understandings about how to ensure SR findings are robust and relevant to policy decision-makers’ questions, have increased the work required. Continued increases in available published evidence have also increased the volume of research included and analysed in SRs.7 We suggest that there are two key factors that have significant time implications for reviewers today.

While historically SR questions were designed to be narrow, the Cochrane Handbook recognises that ‘Increasingly, reviews are becoming broader, aiming, for example, to identify which intervention—out of a range of treatment options—is most effective, or to investigate how an intervention varies depending on implementation and participant characteristics’.8 Review methods have evolved, resulting in a rich diversity of SRs in relation to the types of questions, review approaches and research methods.9 In figure 1, we provide examples to characterise different types of reviews that explore intervention effectiveness and the questions they seek to answer.6 Although we focus on reviews addressing intervention effectiveness, the breadth and complexity of reviews addressing other types of question is also likely to vary.6 10

Variation in the breadth and complexity of reviews addressing intervention effectiveness.24–27

There are numerous factors that may impact on the time required to complete broad, complex or multicomponent reviews and thus increase their risk of remaining unpublished. Below we discuss six key factors.

First, the increased breadth and depth of these reviews inevitably increases the work required. As figure 1 illustrates, broad effectiveness reviews cover more ground than simple effectiveness reviews. They may also require comparative work to assess the relative effectiveness of different intervention types. Thus, while the Cochrane Handbook recommends that searches are ‘as up to date as possible’, specifying within 12 months but ideally within 6 months of publication, it also acknowledges that reviews with a broad scope will ‘require more resources’.11

Complex effectiveness reviews often require significant conceptual development and interpretation, which can be time-consuming. The purpose of these reviews is to understand the critical components and mechanisms of multicomponent or multilevel interventions.12 Since methods to examine intervention complexity are relatively recent, this work also often requires methodological innovation, which further increases the time required.

Multicomponent approaches are often designed to understand intervention mechanisms and contextual complexity in addition to intervention effectiveness. As such, these approaches, such as mixed-method research synthesis or realist synthesis, often involve multiple synthesis stages.

Second, these types of review may also increase the extent of work required for effective patient and public involvement (PPI), which is vital to ensure that SRs produce useful findings.13 PPI can advance fields of study and help SRs address real-world questions from patient-based perspectives.14 However, the extent of PPI undertaken in any review will necessarily vary. As illustrated in figure 2, PPI is typically undertaken at the beginning of a review to identify relevant questions to answer. Further input during the synthesis stage and beyond can help to interpret emerging findings and disseminate findings to interested audiences. Input at the synthesis stage can assist with interpretation of the findings and is therefore particularly useful when broad questions are asked, or when conceptual development is a feature of the review. For example, in broad reviews examining multiple interventions, PPI can help interpret findings on effectiveness by providing insights about the likely acceptability of the various options reviewed. As such, the breadth and complexity of reviews and the extent of PPI required may be positively correlated.

Review procedures and impact on review completion time. GRADE, Grading of Recommendations, Assessment, Development and Evaluation; PPI, patient and public involvement; SRs, systematic review.

Third, to ensure SRs also address practice and policy interests, engagement activities may extend beyond the bounds of PPI and can include consultation with practitioners, research commissioners, policy makers and academics. At the EPPI Centre, we have found it helpful to engage with policy commissioners throughout the review, to understand how emerging findings may be applied. Involvement of policy makers and other stakeholders alongside patient and public stakeholders, will likely increase the review’s relevance and utility, but involving multiple stakeholders will inevitably increase the time required to complete a review.

Fourth, as the SR community continually strives to enhance and improve review methods, the steps required for any review have increased. Figure 2 illustrates several recently recommended review procedures for improving the quality and usability of SRs.15–18 While such advances are welcomed, these additional steps inevitably increase the time required for completion, even for single synthesis reviews. However, for broad or multicomponent reviews, this extra work may be multiplied. For example, in a multicomponent review containing both an effectiveness synthesis and a qualitative evidence synthesis, both Grading of Recommendations, Assessment, Development and Evaluation (GRADE) (a tool for summarising confidence in effects of interventions by outcome across studies)16 and GRADE-CerQUAL (for assessing the confidence of evidence from reviews of qualitative research)17 may be applied. In a broad review using GRADE, it will need to be applied to the findings about each type of intervention examined.

Fifth, as the extent of available research literature continues to grow,7 this can result in larger numbers of studies to screen for inclusion in each review. The impact is exponential for broad or multicomponent reviews. Though there are tools, such as machine learning and priority screening, that can reduce screening time, these technologies are unlikely to significantly impact on reviews that require iterative searching or that include ‘views studies’, which are more challenging to identify.19

Sixth, the report and publication process can result in additional time challenges. For example, sourcing peer-reviewers with expertise in complex reviews and novel synthesis methods, may take considerable time. At the EPPI Centre there have been occasions when it has taken 6 months to receive completed peer reviews, due to a lack of peer reviewers qualified to comment. Also, policy commissioners often require reviews to be written up as project reports and peer-reviewed prior to being written up for journal publication.

As broad, complex and multicomponent reviews will inevitably take longer, the risks of being out of date, and therefore, misleading, may appear to be increased. However, we argue that the risks are less applicable to these types of reviews.