The majority of Never Events (62.32%) occurred in six main departments: General Surgery, 19 (18.81%); Gynecology, 17 (16.83%); Orthopedics, 16 (15.84%); Cardiac and Cardiothoracic, 15 (14.85%); Ophthalmology, 8 (7.92%); and Urology, 7 (6.93%) (Table 1). Therefore, our analysis focused on Never Events’ occurrence in these six departments.

Table 1 Characteristics of the dataset according to surgical specialty Table 2 Characteristics of patients and surgeries in the dataset

To evaluate our models, we adopted the area under the curve (AUC) measure. This measure is especially suited for imbalanced data, as was the case in this study, as it does not have any bias toward models that perform well on the minority of majority classes at the expense of the other [26]. Our three random forest models each demonstrated good performance, exhibiting an AUC between 0.81 and 0.85. Generally, AUC scores between 0.8 and 0.9 are considered excellent [27]. AUC is interpreted as the probability that our model will rank a randomly chosen positive instance higher than a randomly chosen negative one [28]. As such, our models can be considered relatively strong and accurate, despite their limitations.

Feature importance

Figure 1 shows the most common contributing features to the occurrence of Never Events (of both types combined) in the six departments, along with the associated probability change.

Fig. 1

Top 15 contributing features for the six examined departments

The top 14 contributing features varied significantly across departments, and no single feature set was consistently more informative across all operations for predicting Never Events. For example, feature [C], “Discrepancy in second count,” varied significantly across departments (160% to 1,950%). Feature [B], “Surgery is paused because of discrepancy in third count,” appeared in four of the six departments, and the associated probability change varied dramatically, between 269% and 1,540%. There were 10 features that consistently decreased the chance of a Never Event occurring, including [F] “Surgeon scans the cavity/fascia before closure during the second count,” which affected five out of six departments and was consistent in its probability change, between 65 and 100%. Features [I], [J], [ K], [L], [M], and [N] decreased the chances of Never Events between 2 and 100% in three departments. Three features, [A] “Discrepancy in absorbing materials,” [E] “Surgery time > 4 hours,” and [G] “Surgery time < 1 hour” appeared once across departments, with a medium impact on Never Event occurrence.

Analysis of the results by department shows variation among the contributing features. For example, in Ophthalmology, the probability was consistently − 100% for five features, while in General Surgery, two features that increased the probability of an error varied between 1,168–1,283%: features [B] “Surgery is paused because of discrepancy in third count” and [C] “Discrepancy in second count.” In Orthopedics, those same two features, [B] and [C], increased the probability of error (1,540–1,950%). Three features decreased the probability of error: [F] “Surgeon scans the cavity/fascia before closure”; [H] “Second count is performed before closure of fascia/cavity”; and (I) “Procedure type is compared to the one written in patient’s file,” by -65 to -87%.

Effects of feature combinations

In the following analysis (Fig. 2), we examine the effects of paired features, i.e., features that occur together in the data. It is important to note that when considering feature combinations, their occurrence is expected to be very low, especially in the Never Events class. As such, the estimated effects are likely to be very high, yet their confidence is significantly low.

Fig. 2

Effect of two features’ combination on prediction by surgical departments

Interestingly, in General Surgery, there were 14 feature combinations that caused a probability change of 13,600% (Fig. 2A). In comparison, the single feature analysis (Fig. 1) revealed a probability change of 1,287% and 1,168%, surprisingly by two features that were not part of the 14 feature combinations identified here.

In Fig. 2A (Gynecology), the effect of every feature combination is associated with a probability change of 1,000–2,000%. In the single feature analysis (Table 2), the effect of two of the features separately was < 900%, and the rest lagged behind with < 150%. In Urology (Fig. 2B), the results showed there were dozens of pairs with an effect of 1,900–2,500%, while the effect of a single feature had < 1,150% effect on error. In General Surgery (Fig. 2E), the accumulated effect of two features together showed a dozen pairs with an effect of 1,900–4,200%, while the effect of a single feature had a < 1,950% effect on error, and the rest showed even lower percentages.

Features affecting types a and B

Turning to Models 2 and 3, there was an overlap in three of the top five contributing features to type A and B errors (Figs. 3 and 4): (1) the presence of two nurses during the surgery predicted a greater occurrence of type A (66%) and type B (88%); (2) an operation < 1 h had a greater occurrence of type A (122%) and type B (87%); and (3) when the operation lasted between one to two hours, both types A and B were less frequent, decreasing by 60% and 74%, respectively. The surgical department that was most affected regarding the occurrence of type A Never Events was Ophthalmology, with a prevalence of 504%, while General Surgery was associated with a decrease of 63% in type A (Fig. 3). For type B, the two remaining features were staff driven; the feature “more than three physicians” was associated with an increased prevalence of type B (151%), while “two physicians” was associated with a decreased prevalence of Type B, by 52% (Fig. 4).

Fig. 3

Features affecting the wrong surgery site (type A)

Fig. 4

Features affecting retained foreign items during surgery (type B)