Today, any organization requires innovative, flexible solutions to digitize and automate their processes [2] in conjunction with new technologies [3] for increasing their competitiveness and productivity [4], [5]. In healthcare environments, digitization has greater disruptive potential because it affects aspects like patient care, spiraling costs, quality and rewarding value [6]. However, before the technological boom of the last decades, patient care was usually based on the manual application of clinical practice guidelines1 (CPG); i.e., it was usually based on paper-based medical reporting without automatic support (such as computerized systems for clinical decision support).

Today, many authors have studied the benefits of establishing well-defined CPG digitization processes2 to both healthcare professionals and patients [7], [8], [9]. CPGs help reduce variability in clinical practice and improve the quality of clinicians’ performance and decision-making. CPGs pool existing knowledge to facilitate the use of effective, reliable interventions based on empirical evidence and clinical experience [10], but the CPG digitization processes make this effort more efficient and reliable by using computerized clinical decision support systems.

Over the last decade, many scientific initiatives and technological proposals have been published to facilitate the automation and digitization of clinical guidelines. These technological initiatives are referred to as CPG-derived CDSS (clinical decision support system) in this paper. Most of them, however, have been limited in their functional scope, or their practical application has focused on treating specific pathologies in controlled environments. After an initial rise of formalisms and languages [11], it is necessary to assume and address the fact that the actual applications of CPG-derived CDSS are limited. This situation has been analyzed by several authors from different perspectives and the reasons for this are heterogeneous [12], [13], but the reasons are usually related to clinicians’ limited understanding and trust in the underlying models, thus creating poor engagement in their usage [14]. In addition, other barriers have been identified in the scientific literature [15]: limitations for the integration of these systems into EHR (Electronic Health Record) systems and clinical workflows; lack of sufficient patient-specificity; mismatch to the cognitive tasks and processes of the end user (healthcare professional); lack of change management mechanisms in clinical recommendations and clinical processes in runtime; and effective interoperability mechanisms.

In this context, digital health innovations related to CPG have not been adopted on a large scale and are usually abandoned when they are not upscaled or kept in use over time at organizational or system level [16], [17]. As mentioned above, the factors influencing non-adoption and abandonment are complex and include health conditions, technology, value propositions, adopters’ systems (professional staff, patients, and lay caregivers), organization(s), institutional contexts, as well as interaction and mutual adaptation between these factors over time [16], [18]. Consequently, the success of the CPG-derived CDSS implementation will clearly depend on the analysis of previous critical success factors, and modeling efforts should allow for the broadest and most effective use of these systems based on models of technology adoption, evidence-based practices, and conceptual models in clinical practice [15].

The contributions of this paper. The final purpose of our work is to support decision makers with a method based on quality models that provides stakeholders with information regarding the eventual adoption of a new technology into their organizations according to their objectives. For this purpose, this paper describes a method based on quality models to uniformly compare and evaluate technological tools, offering a rigorous method that uses qualitative and quantitative analysis of technological aspects. Later, this method is instantiated to evaluate and compare five currently available and specific CPG-derived CDSS (GLEE, ArdenSuite, GLARE, DeGeL, and KnowWe) by highlighting each phase of the CPG digitization lifecycle. The opinion of technology consultants who are experts in the application of information and communications technology (ICT) in the healthcare environment were considered for this purpose.3 This evaluation is carried out objectively and uniformly testing each technological tool on each technological aspect included in our quality model. After carrying out this evaluation, the final support score of each CPG-derived CDSS is obtained applying the method described in this paper.

Table 1 summarizes the contribution of our paper to the existing literature considering the problem, what is already known and what this paper adds.

The rest of the paper is organized as follows. Section 2 presents the methods used in this paper; specifically, it describes the phases that make up this method, as well as the quality model (c.f., Section 2.2.2) and scoring method (c.f., Section 2.2.3) that was used to evaluate each tool under study. Subsequently, Section 3 presents our results and how our quality model on CPG-derived CDSS has been applied and what results of its evaluation have been, respectively. Later the discussion and analysis is presented in Section 4. Finally, Sections 5 Related works, 6 Conclusions and future works describe some related works and conclusions as well as future work, respectively.