We test the association of hospital process digitalization measured in three different ways with two process and two outcome quality indicators. Our main model results do not show a consistent digitalization-quality relationship across indicators. Regarding the two process quality indicators, all but one of the associations were insignificant. For mortality pneumonia, we found some evidence that higher (process) digitalization is associated with better outcomes. Although statistically significant, the magnitude of this association is rather small: A one-point increase in the total DR-score, scaled between 0 and 100, is associated with a 0.004 points lower mortality ratio. For mortality pneumonia, this negative association is not consistent when considering the sub-dimensions: The association with the sum of sub-dimensions is statistically insignificant and we find both positive and negative associations with single sub-dimensions.

Regarding new decubitus cases, greater process digitalization was associated with worse quality, albeit rather weak in magnitude with regards to models (1) and (2). Still, this finding is already visible in the correlation analysis and consistent across all three model specifications. Model (3) shows that the sub-dimension documentation/ diagnosis seems to account for a large part of the positive association as it is both strong in magnitude and statistical significance. This might hint at the fact that the observed positive association could in fact not be due to actually worse quality, i.e., a higher number of unexpected new cases developing decubitus, but rather a better diagnosis, documentation, and reporting of decubitus cases. This is supported by the results of the second sensitivity analysis including quality indicator value outliers as with the inclusion of these qualitatively worst performing hospitals, statistical significance weakens strongly, and most associations become insignificant in the multivariate analysis.

We discuss our findings along conditions (2) and (3) specified in the introduction (for why we believe we comply with condition (1), i.e., digitalization indicators must be capable to measure variation in hospital (process) digitalization, see Data – DigitalRadar Score in the supplementary material). Regarding condition (2), namely the statistical sensitivity of the quality indicators used, our descriptive results show that there are many observations of “perfect” quality (i.e., values of 0). Moreover, observations of low quality (values greater than 15 for the process quality indicators; values greater than 1.0 and especially 2.0 for risk-adjusted ratios) are scarce. When including quality indicator outliers (i.e., very low quality) in our robustness check, our findings do not change. Therefore, we suspect that the quality indicators cannot detect quality differences between hospitals sufficiently well – at least not in the context of our research questions. The esQS program is a quality monitoring program that was developed to detect the worst hospitals in order to engage them in structured quality dialogs [23]. Moreover, all quality indicators were self-reported. These two facts might explain why we do not observe many examples of (very) low quality, affecting our ability to find statistically significant associations between process digitalization and hospital quality.

Condition (3) implies that the causal pathway between digitalizing processes and the investigated quality indicators should be explained. For transparency, we provide one example of our selection and matching logic. For instance, we argue that preoperative waiting time of hip replacement and osteosynthesis surgery patients is linked to the DR-question “In the emergency room, patient admission, triage, medical orders, and documentation tasks are carried out digitally. This is done via the [hospital information system] or special systems with interfaces to the [hospital information system]” [22]. We hypothesize that digitalization of these emergency room processes (admission, triage, etc.) might lead to faster decision-making, decreasing the preoperative waiting time for emergency and urgent surgeries, including hip replacement and osteosynthesis surgery after femur fracture which classify as urgent surgeries. Thus, we included the question’s sub-dimension (i.e., documentation/ diagnosis) in models (2) and (3).

While our assumptions regarding the digitalization-quality mechanism might be valid, we believe that a problem might arise from how process quality is measured in the esQS program. While process digitalization might decrease pre-operative waiting time measured in minutes or hours, the esQS quality indicators measure the share of cases receiving surgery later than the pre-defined threshold of 24 h after admission, which is much harder to affect with process digitalization.

Martin et al. analyzed the association of the NHS Clinical Digital Maturity Index (CDMI), scaled between 0 (not digitalized) to 1400 (fully digitalized), and its three dimensions, readiness, capability (most closely related to process digitalization), and infrastructure, with five different quality measures [13]. For the 30-day hospital level mortality index, the percentage of episodes of care with complications, and the number of emergency readmissions within 28-days of discharge, the authors found no significant associations. In contrast, risk-adjusted long length of stay was positively associated with the CDMI score while a higher CDMI score was associated with more harm free episodes of care. Overall, these findings are in line with our findings.

Van Poelgeest et al. examined the relationship between EMRAM stages and the “Elsevier best hospitals score”, a composite measure based on 542 quality indicators, as well as the overall Elsevier score’s three domains – medical care, patient orientation, and effective treatment [15]. The authors only find insignificant relationships which might be evidence for the importance to fulfil the conditions we defined for investigating a digitalization-quality relationship. With respect to condition (3), the authors do not provide specific hypotheses for a match of their digital maturity measure and quality indicators. Regarding condition (1), the EMRAM stages are ordinally scaled from 0 to 7, and a hospital can only advance to the next stage if it complies with all the requirements of the former stage [20]. Thus, these stages can detect major differences only in digital maturity milestones rather than processes’ digital maturity. Finally, concerning condition (2), the authors used “bundled” values for the quality measures, in the form of an ordinal scale from 1 to 4 for composite measures comprising lots of different indicators. Detecting quality variation between hospitals with such quality measures might be difficult in the context of investigating a digitalization-quality relationship.

Motivated by the above findings, we specified three models in our study changing the explanatory variable for hospital process digitalization. Martin et al. argued, for instance, that the inconsistent relationship they found might be due to the fact that the CDMI score and its dimensions did not measure all quality-relevant aspects of digitalization. We suspected the same regarding our model (1), i.e., when only considering the total DR-score. We aimed at alleviating this issue by including specific aspects of process digitalization in our models (2) and (3). As sub-dimensions are measured between 0 and 1, our models (2) and (3) might in turn have drawbacks regarding condition (1) for showing a digitalization-quality relationship, i.e., while describing process digitalization at a detailed level, they might not measure variation between hospitals sufficiently.

With longitudinal data, within hospital changes rather than between hospital differences could be assessed, e.g., with fixed effect models, addressing the challenges discussed above. Such data will become available after the second measurement period of the DR evaluation project will have been completed in the second half of 2024. Furthermore, future research could assess the relationship between process digitalization and measures commonly used in intervention studies (e.g., [27]) such as time saved in a process, reduced waiting time, or changes in utilization. However, finding nationwide data of such measures might be challenging. Thus, we argue that when developing national programs to measure hospitals’ digital maturity, suitable effectiveness measures, e.g., routinely available in hospital information systems, should be collected along with digital maturity data.

All esQS quality indicators were self-reported. This might create reporting bias. Similarly, the DR-score is based on a questionnaire; i.e., it is also a self-reported measure. Still, reporting bias might be rather small for the DR-score as hospitals had an incentive to answer questions truthfully [21]: After participating in the evaluation, hospitals could use a benchmarking tool allowing them to compare their own digital maturity to that of their peers. If questions were deliberately answered incorrectly, this tool would become worthless for a hospital. In addition, results were checked for plausibility and completeness, further alleviating concerns regarding reporting bias (cf. Data – DigitalRadar Score in the supplementary material).

We included a one year time lag in our main model. Process and outcome quality might be affected by digitalization only in the mid- to long-term, however. One reason for this is because digitalization of structures and processes needs to be accompanied by educating and training staff and process owners must implement and get accustomed to optimized, digitalized processes. Unfortunately, in the DR-dataset, we cannot identify for how long a certain structure or process had already been digitalized. New measurements and longitudinal data could also help with this issue (cf. above).