NICU antibiotic stewardship efforts have been underway for several years, but EOS-related antibiotic exposure still far exceeds rates of proven EOS [5]. Participation in this collaborative led to a collective 15.3% reduction in mean aggregate AUR without evidence of increased EOS or NI rates within individual participating NICUs. The AUR reduction was below the pre-specified aim. However, this study provides helpful antibiotic stewardship lessons for individual prescribers, NICU leaders, and multi-site QI efforts. This is the first multicenter NICU antibiotic stewardship collaborative that we are aware of reporting comprehensive monthly AURs enabling accurate assessment of improvement and sustainability. Approximately one-third of NICUs met the goal of reducing mean AUR by >25% and sustained the reduction until the end of the collaborative. Nearly half of the sites reduced AUR by >25% but did not sustain the AUR reduction. More than half of the NICUs did not meet the goal of >25% reduction. In 2021, 31 California NICUs joined a federally funded antibiotic stewardship quality improvement collaborative [33]. Most of them joined because they had opportunities to improve their antibiotic use rates. NICUs still have a sense of urgency to improve appropriate antibiotic use in NICUs. We explored the variable range of improvement in this study to provide helpful lessons for current and future NICU antibiotic stewardship efforts and studies.

Special cause variation, with reduced aggregate AUR, was noted during the third month of the collaborative. Each subsequent monthly AUR of the 18-month collaborative was below the pre-collaborative mean AUR. Although EOS antibiotic use does not reflect all antibiotic use in NICUs, the achievement of sustained improvement by one-third of participating NICUs in our collaborative suggests that EOS may be a reasonable area to begin stewardship efforts. However, a relatively small proportion of participant NICUs contributed to the collective improvement. Nearly half of the participating sites showed early improvement but were unable to sustain improvements during the relatively short time frame. NICU stewardship efforts should focus on addressing sustainability early and integrate processes into the standard work to increase the likelihood of maintaining improvement.

We visualized individual NICU monthly AUR charts, individual NICU range of performance, and contributions to the aggregate performance by displaying all NICU monthly AURs in a small multiples figure (Fig. 3). The figure highlights the challenge of wide monthly variability in identifying improvement. The display is an important supplement to accurately describe aggregate data as a product of individual NICU context and performance. We attempted to identify context and process factors associated with higher-performing NICUs. Leveraging the wide range in degree of improvement among the sites, we analyzed qualitative survey data to identify factors associated with higher-performing NICUs. Higher performing teams, with AUR reductions >20%, identified multidisciplinary collaboration, leadership, and frequent antibiotic usage review as primary drivers of improvement. However, our quantitative analysis of characteristics did not identify more specific context characteristics or processes associated with higher-performing NICUs. Participation in the collaborative may have only facilitated improvement in NICUs with local factors supporting readiness for improvement. Notably, one site (NICU # 13), showed an increase in AUR during the collaborative. This site reported their specific challenges during the last study webcast. When asked specifically about perceived contributors to the increased AUR, they reported continuation of antibiotics beyond seven days due to abnormal labs, lack of documentation of intent to discontinue antibiotics in initial admission assessment documentation, high patient acuity, and staffing challenges that limited daily antibiotic rounds and antibiotic review meetings.

Concern for missed infections, morbidity, and mortality related to delays or early discontinuation of antibiotics was a persistent concern noted throughout the study. Overall, this study did not detect increased annual EOS or NI rates related to the study period. Annual data showed that one site had an increase in all-cause mortality rate during the collaboration. However, as with all the other NICUs, this NICU did not have an increase in annual EOS or NI during the years of this study. Although more large-scale studies exploring the safety of antibiotic reductions are needed, these data support the safety of antibiotic stewardship in a large group of NICUs with AUR reductions over time without infection-related morbidity and mortality.

A cross-sectional study of antibiotic exposure in full-term and late preterm infants during the first week of life (among 11 countries in Europe, United States, and Australia, including the year after our study, through 2018) documented antibiotic exposure for EOS that is disproportionately high relative to rates of EOS [5]. They also noted high variability with a 9-fold difference in antibiotic exposure among sites. Given this recently published data, there is still a lot to learn about sustaining antibiotic use that is more consistent with rates of proven infection. Our study was conducted before the 2018 AAP guidelines for EOS in neonates were published [34, 35]. However, our study provides several practical lessons to address the continued overuse of antibiotics.

This study provides three important stewardship lessons for future large-scale NICU stewardship efforts and individual sites. Firstly, one NICU retrospectively stratified their AUR by isolating term infants and noted there was special cause variation with reduced AUR in this subgroup analysis that was not detected with AUR, including all gestational ages. Relying on AUR that includes all babies and all patient days may overlook important improvements that could signal initial success with a process change targeting a specific subgroup [36]. NICUs may benefit from stratifying antibiotic use measures by the specific patient characteristics and by time frame targeted by a given stewardship strategy to support sustained improvement. Secondly, as noted, our qualitative results highlighted frequent review of antibiotic use data as an important driver of improvement. Many sites could benefit from robust systems providing more frequent assessment of statistical process control analysis of antibiotic use. In 2021, none of 30 sites registered for the Optimizing Antibiotic Stewardship in California NICUs collaborative stewardship study were stratifying AUR into subgroups and very few were using automated capture of AUR with frequent statistical process control analysis. Current EMR infrastructure and statistical process control software make this process relatively easy and high value. A delay of 1–2 months to review time series AUR data can hinder stewardship efforts. Designating clinicians to perform this analysis at least monthly and marketing this data widely may be beneficial.

Lastly, individual practice variation in antibiotic decisions was a notable barrier raised by site QI teams. Individual practice variation is repeatedly noted as a barrier to providing optimal care, but is usually not formally addressed during QI efforts. Therefore, we used vignette research methods to identify and describe practice variation among prescribers at participating NICUs. There are wide ranging drivers and individual factors that determine whether and individual prescriber will start or stop antibiotics. The factorial vignette study results, described in a separate manuscript, objectively identified specific individual prescriber decisions as primary targets for further stewardship efforts [32]. The vignettes engaged prescribers who were not directly involved with QI teams, provided an objective description of variation, and allowed individuals to compare their practices among peers. Patient case simulations with feedback to providers have been shown to improve quality of care, compliance with evidence-based practices, and reduce costs [37, 38]. Vignettes may help reduce variation, improve stewardship practices, and sustain more appropriate AURs in NICUs if used at multiple time points during stewardship efforts [39]. Reflecting on our experience with this study, analyzing, interpreting, and sharing results from vignette assessments was an important supplemental collaborative quality improvement tool. Further research is required to determine effectiveness and ideal strategies for implementing vignette methods to support antibiotic stewardship in collaborative QI. However, vignette assessment should be considered for any QI project where practice variation is a notable barrier to improvement.

EMR-based interventions were not correlated with reduced AUR in this study, but EMR-based interventions have the potential to support stewardship [40, 41]. Significant improvements have occurred in EMRs in NICUs over the past 5 years with many sites transitioning to more sophisticated EMR platforms with a range of decision support and processes that could supplement stewardship efforts. The evidence supporting EMR interventions improving stewardship in hospitals is low quality, so the potential impact of NICU specific EMR processes targeting antibiotic stewardship is not known [42].

There was widespread adoption and compliance with antibiotic time outs over the course of the collaborative. As noted, this was one of the change package strategies recommended to sites, while leaving sites the autonomy to create their own specific time out process. For example, some sites adopted a time-out process as part of medical team sign-out. Some NICUs adopted a “third party” approach with a pharmacist directly approaching the medical team when the pharmacist performed their routine check of gentamicin levels around 48 h. In 2014, the CDC recommended a time out process within the core elements of antibiotic stewardship [22]. The 2018 AAP guidelines for the management of EOS, published after our study, recommend discontinuation of antibiotics at 36–48 h if there are negative blood cultures and no indication of site-specific infection [34, 35]. Our study supports that large scale implementation of antibiotic time out processes can be sustained over time. However, wide-scale successful implementation of EOS antibiotic time outs alone did not translate to the desired improvement for many sites. Thus, time outs may be one of several processes required to sustain more appropriate antibiotic use.

Effective strategies for externally facilitated NICU antibiotic stewardship programs and/or stewardship collaboratives have not been well studied. Dukovny et al. noted reduced antibiotic use in a 146 NICU multicenter antibiotic stewardship collaborative, but antibiotic use was only assessed by four single-day audits. They noted reduction in median AUR from a baseline of 16.7% to 12.1% [23]. Schulman et al.—with some AUR data overlapping with our study—retrospectively studied AURs in California NICUs and showed that NICUs participating in externally facilitated antibiotic stewardship projects had larger reductions in annual AUR compared to non-participants [4]. Our study and these studies suggest that externally facilitated stewardship programs may help optimize stewardship. However, identifying the optimal methods by which quality improvement collaboratives and individual NICUs improve and sustain more appropriate antibiotic use rates requires more rigorous quality improvement studies. These studies may require more detailed information on individual site context, individual prescriber decisions, timing of specific interventions, and longer sustainability follow-up periods.

This study has several limitations. The participating NICUs paid to join, which may introduce selection bias for NICUs with relatively more resources and high levels of motivation for change. Participating NICUs had different characteristics relative to non-participant NICUs in California, which could limit the generalizability of our results. We did not have an adequate control group with monthly AUR data to compare performance among other NICUs. There was a downward trend in aggregate AUR leading up to the start of the collaborative, which limits our ability to ascribe the improvement to the intervention. Although we were unable to collect the pre-planned balancing measures during the study, our retrospective linkage of individual NICU annual infection data provides a more robust balancing measure assessment than our pre-specified and commonly used measures of “restarting antibiotics” and “readmissions” for antibiotic treatment.