This case study found that Ti/O screening performed in accordance with statistical expectations while Ti + Ab performed significantly better than was expected. It should be noted that the Chi-squared statistic only tests whether the screening approach’s performance is statistically different from a composite or average level of performance. It does not indicate how or why there is a difference in performance. Only a comparison of the summary performance statistics from the test and average contingency tables (i.e., specificity, sensitivity, accuracy, and predictive power) can address the questions of how and why there is a relative difference in performance. This case study found the Ti + Ab screening approach at the first-level screening phase had higher predictive power due to being more effective at finding qualified studies to include in the next stage of evaluation compared to the Ti/O screening approach. This suggests that reviewing both the title and abstract during first-level review stage resulted in a more effective screening performance.

Our case study demonstrated that both the Ti/O and Ti + Ab first-level screening approaches achieved high specificity (98.4% and 98.8%, respectively), due to the small base of qualified studies within the total number of citations included at the initial screening level. In our case, high specificity was easy to achieve as the actual prevalence of eligible studies in the initial search results is relatively small (36/8526 = 0.4%) allowing for a high rate of correct identification and exclusion of unqualified studies by both approaches. Likewise, both the Ti/O and Ti + Ab first-level approaches also achieved high accuracy (98.3% and 98.8%).

The Ti + Ab approach’s predictive power of 25.5% was observed to be higher than the Ti/O approach’s power of 14.3%. This suggests that more information assessed at the first-level screening will likely increase the number of studies correctly predicted to be eligible; thus less time will be required for the next stage of evaluation. This also suggests that despite the Ti/O approach including comparably more studies for the next stage evaluation compared to the Ti + Ab approach (154 citations versus 141 citations, respectively, Table 2), the predictive power of the Ti/O screening approach is still lower and thus less reliable for identifying qualified studies. Further, compared to the Ti/O approach, the Ti + Ab approach had a much lower excluded “trues” rate (5.3% vs. 41.1%) and higher sensitivity (57.9% vs. 94.7%) highlighting that the latter approach is more likely to capture studies of interest and less likely to pass unqualified studies to the full-text level, potentially saving both time and financial resources. Methodologies that systematically lead to missing relevant articles result in evidence selection bias, a bias that occurs when all available data on a topic has not been identified [16]. This can impact the synthesis of evidence and bias the resulting conclusions, in a direction inconsistent with the true association [2].

Our findings parallel a previous study which compared the differences between Ti/O and Ti + Ab screening approaches where Mateen et al. [8], found that Ti + Ab screening achieved higher precision compared to the Ti/O approach and required the review of fewer full-text articles compared to the Ti/O screening. However, the authors also suggest that Ti/O screening was possibly more efficient than Ti + Ab screening because of the expected time saved from not reading the abstracts of unqualified studies, even though more time was required during the full-text review stage. The authors do admit that they did not measure relative screening times and thus could not provide substantiation of their time-savings hypothesis.

Choosing a screening approach may be based on the nature of the eligibility criteria. Titles alone may not have enough information to make predefined PICOS eligibility judgments. More detailed eligibility criteria may increase the likelihood that titles alone will not be enough to reveal if a study qualifies. Intermediate markers of sarcopenia risk (i.e., skeletal muscle mass, muscle strength, muscle performance) may not have been explicitly mentioned in titles but only relayed in the abstract or the full text. For example, in this case study, four titles that mentioned “frailty” were screened out using Ti/O because this was not an outcome of interest but the Ti + Ab screening provided the opportunity to review more detailed methodology and information regarding measurement of secondary outcomes that were consistent with the eligibility criteria. Information in the abstracts revealed that frailty can be determined by a variety of outcomes including gait speed and hand grip which were also eligible measurements of muscle strength and muscle performance in our protocol (see Additional file 2). Zhu et al.’s “A Prospective Investigation of Dietary Intake and Functional Impairments Among the Elderly [17]” does not mention investigating impairment in walking capability, a measure of muscle performance, in its title but this information is available in the abstract. To capture these outcomes of interest, Ti + Ab screening would become a necessary next step of Ti/O screening, and erring on the side of inclusion to screen more full texts may be needed for eligibility determination. There are situations when the title or sometimes even the abstract of a paper does not contain all the PICOS information necessary to predict whether the citation should be included. This is particularly true for studies published previous to reporting guidelines: the CONSORT Statement [18], the reporting guidelines for randomized controlled trials was developed in 1996; and QUORUM, the predecessor to the PRISMA Statement, the reporting guidelines for systematic reviews and meta-analyses was developed in 1999 [19]. In the current review, the information needed to assess whether a dietary pattern based on a macronutrient distribution, where at least one macronutrient had to be outside of AMDR, would often not be present in the title alone (e.g., “Adult macronutrient intake and physical capability in the MRC National Survey of Health and Development [20]”) nor sufficiently specified in the abstract. In this case, erring on the side of inclusion for full-text review during screening would be a viable strategy to avoid missing relevant citations.

The Ti + Ab approach to first-level screening in this study mistakenly excluded two citations, one per Ti + Ab reviewer, that were ultimately included in the final analysis, despite having access to the abstract. Due to the complexity of the eligibility criteria in this case study, it can be debated that each citation should have been reviewed in tandem per screening approach to avoid excluded trues. It is common for teams to debate whether some studies fit the eligibility criteria, even at the full-text level, as in the case of these two studies (see Additional file 2).

Should Ti/O prove effective and time-saving as a screening approach, reporting guidelines should consider recommending study titles to include all PICOS elements, as currently the abstract is used for this information. Implementing revised reporting guidelines would further require reconsideration by journals of title character limitations which currently prove a challenge when attempting to include all PICOS elements. Currently, both the current CONSORT Statement [21] for randomized controlled trials, and the PRISMA Statement [22] for systematic reviews and meta-analyses recommend that the study design is explicit in the title. CONSORT explicitly requires PICO to be included in the journal and conference abstracts [23] and the PRISMA Statement recommends that the study design is explicit in the title and that the eligibility criteria (which usually reflects PICO) are reported in the abstract [22]. Additionally, the Journal of the American Medical Association as well as the Annals of Internal Medicine requires the type of study design (i.e., clinical trials, meta-analyses, and systematic reviews) as part of the publication’s title [24, 25]. While not an exhaustive review of the topic, these examples suggest that standard reporting guidelines and journal requirements for titles may not readily support the use of the Ti/O screening approach.

This case study has limitations. First, it would have been more methodologically optimal to have two investigators conducting Ti/O first-level screening in tandem for consistency with the Ti + Ab first-level screening. Further, there were differences in backgrounds in investigating teams for the two approaches: the Ti/O approach was performed by an investigator with subject matter expertise while Ti + Ab approach was performed by two investigators with systematic review expertise. Second, the average time to conduct the first-level screening using either approach was not measured. This would have been useful information as time taken is essential to reflect both efficiency as well as cost. Time was not tracked as this case study was a secondary focus to the original systematic review. This being said, we would recommend tracking time in future similar investigations. Lastly, this case study is based on the manual screening of citations and may not be generalizable to screening approaches using artificial intelligence (A.I.) or automated screening technology (e.g., DistillerSR or Covidence) to lessen the workload attributed to screening. On the other hand, this case study has strengths. First, this case study utilizes eligibility criteria developed by (although endorsement by not implied) the United States Department of Agriculture’s Nutrition Evidence Systematic Review team (NESR) [10, 26, 27], government experts that specialize in conducting food- and nutrition-related systematic reviews. Most recently, NESR implemented a Ti/O approach at first-level screening for 33 original systematic reviews conducted to support the 2020 Dietary Guidelines Advisory Committee, which informed the development of the 2020–2025 Dietary Guidelines for Americans [28, 29]. Second, the results of this case study are supported by two systematic reviews that have both been recently peer-reviewed and published [10, 26]. Lastly, this case study adds to the limited knowledge base of evaluating the difference in Ti/O compared to Ti + Ab screening approaches when conducting systematic reviews.