The research proposal for this project has been published elsewhere [10]. Changes made to this proposal during the project are described in Supplementary Material 1. Ethical approval for this research has been obtained from the Ethics Committee from the Faculty of Behavioural, Management and Social Sciences of the University of Twente (request number 221418).

3.1 Design

To formally include Open Science within ASHEA, the learning objective ‘Apply open-source modelling practices in the context of health economic modelling’ was added to the already-existing learning objectives of the course.

To design the teaching innovation, an electronic survey was sent to HEOR professionals and previous ASHEA course participants (N = 22) to determine which aspects of Open Science and OSM this teaching innovation should focus on. The survey was designed by XP and was tested for comprehensiveness and completeness by a PhD student in HEOR (Supplementary Material 2A). The survey was spread among HEOR professionals through a LinkedIn post and through a HEOR professional newsletter in the Netherlands. Previous ASHEA course participants were sent the survey via email. The first part of the survey asked respondents to mention the length of their professional career, their work environment and whether they develop and/or review health economic (HE) models as part of their daily work activities. The second part of the survey asked the respondents which Open Science-related skills they believe should be taught to future HEOR professionals. Responses to this second part of the survey were used to decide on which topics the teaching innovation should focus. Finally, respondents could add general feedback and comments on the survey and on teaching Open Science in general.

Four previous ASHEA course participants (18%) and 19 HEOR professionals returned the survey. Due to the limited number of respondents, survey results from both groups were combined and are discussed together. Most respondents worked within a consultancy (N = 9, 39%) or academic environment (N = 8, 35%). Thirteen (56%) had professional career of 5 years or shorter. Most respondents neither developed (N = 15, 65%), nor reviewed (N = 14, 61%) HE models during their daily work activities (Table 1). According to the 23 respondents, the most important OS-related skills were (1) being able to use a script-based statistical software, (2) script annotation and documentation, (3) sharing software code, (4) communicating your work to a lay audience, and (5) publishing open access (Table 2). Based on these results, the teaching activities described in Table 3 were developed. The slides of these teaching activities are available at the following link https://doi.org/10.5281/zenodo.8214973 for interested (HEOR) teachers who may want to include these activities in their own teaching. Table 3 also contains references to selected relevant resources that may be used to design teaching activities on these Open Science topics.

Table 1 Demographic characteristics of the respondents to the survey on important (HEOR) Open Science-related skillsTable 2 Most important Open Science-related skills for future HEOR professionals according to survey respondentsTable 3 Overview of the developed teaching activities3.2 Implementation

The teaching innovation started with a 2 h lecture introducing the rationale behind and the concept of Open Science. Additionally, this lecture introduced course participants to Open Science and OSM-related initiatives within HEOR. All other topics were introduced during shorter lectures of 15–30 min at the beginning of the hands-on tutorial sessions that were scheduled during the course. At the end of the course, the student pairs had to present how they would design and evaluate a public outreach activity to disseminate the process of developing their HE model and their results to a non-scientific audience. Each pair could choose which public outreach activity to design from the following list: developing a R shiny interactive application, a game, an infographic, a podcast, a TED-talk-like presentation, social media posts, a blog, or a video. The best presentation was elected by the students, according to criteria that they had formulated themselves. These criteria were gathered by the teaching staff before the presentations and communicated to the students before the start of the presentations. The criteria that students formulated to elect the best presentation related to the content and the form of the presentation, and to the impact on the societal behaviour of the public outreach activity. The content-related criteria were that the content should be (a) concrete and directly applicable in practice and (b) complete, thus containing all relevant aspects. The form-related criteria were the clarity of content, the attractiveness of the presentation, the duration, the engagement with the audience and whether there was a demonstration of the public outreach activity.

To formally assess whether participants achieved the learning objective related to Open Science, pairs of students had to write a reflection on Open Science in relation to HE modelling and their project in the final project report. This reflection had to include a description of how participants had included Open Science practices during their project and describe its importance in the HE context. Besides, participants had to select one Open Science practice that they thought was less valuable for HE modelling and motivate their choice. The actual implementation of the Open Science practices in their project was not graded separately.

3.3 Evaluation

The effectiveness of the teaching innovation was assessed using multiple methods. Firstly, a survey was distributed to participants (N = 10) at the beginning and end of the course to assess their knowledge of Open Science (Supplementary Material 2B). Participants were asked to define Open Science. We assessed whether the participants’ definition contained the following Open Science concepts mentioned in the United Nations Educational, Scientific and Cultural Organization (UNESCO) recommendation on Open Science [1] (Box 1):

Openly available,

Openly accessible,

Reusable,

Increase collaboration,

Sharing information (with society),

Open the processes of scientific knowledge creation (transparency),

Open the processes of scientific knowledge evaluation,

Communication to societal actors beyond the traditional scientific community,

Open scientific knowledge,

Open science infrastructures,

Science communication,

Open engagement of societal actors.

Box 1: UNESCO’s definition of Open Science

In the same survey, attitudes of participants towards scientific knowledge and the participants’ opinions concerning researchers’ compliance to Open Science philosophy and practices were also elicited before and after the teaching innovation using five-point Likert scales (Supplementary Material 2B; Figs. 2, 3). Participants were asked to rate their proficiency level concerning Open Science-related skills before and after the teaching innovation using the following scale: ‘No knowledge at all’, ‘Basic knowledge’, ‘Advanced knowledge’, ‘Expert’. The included Open Science-related skills were the same as those included in the survey distributed among HEOR professionals and previous ASHEA course participants (Table 2). The survey was concluded with an open field where participants could write general feedback and comments. Frequency plots of the participants’ answers to the Likert scales were drawn to assess the participants’ attitudes towards Science, their opinions concerning researchers’ compliance with Open Science principles and their self-reported proficiency level in the different Open Science-related skills.

Secondly, informal group discussions to evaluate the relevance and quality of the Open Science-related teaching activities were organised after half of the Open Science-related teaching activities had taken place and at the end of the course. XP asked the following questions to the group during these evaluation sessions: ‘I find the selected Open Science-related topics relevant for my professional development’, ‘The content of the Open Science-related activities is too complicated | just right | or too easy’ (multiple-choice question), ‘Are there any Open Science-related topics you are curious about and would like to discuss in a following session?’, ‘Would you like to have more practical assignment to apply these Open Science principles to your own health economic model? (Yes/No)’ and ‘Do you have any further feedback?’. During these evaluations XP took notes which he summarised.

Seven (70%) participants filled in the survey before and after the teaching innovation. When comparing the participants’ definition of Open Science to the UNESCO definition, the following concepts were more often mentioned after the teaching innovation: openly available, reusability, sharing information with society, transparency, communication beyond scientific communities and open scientific knowledge (Fig. 1). The concepts increase collaboration, science communication and open engagement of societal actors were not mentioned in any participants’ definitions.

Fig. 1

Comparison of the proportion of participants’ definitions of Open Science (before and after the teaching innovation) mentioning concepts from the UNESCO definition of Open Science. OS, Open Science; after, after the teaching innovation; before, before the teaching innovation

Concerning the participants’ attitudes towards scientific knowledge, less participants agreed after the teaching innovation compared with before the teaching innovation with the statement that ‘all scientific knowledge is produced according to open and transparent processes’ and that ‘open-source publishing of HE models is common practice’ (Fig. 2). After the teaching innovation, less participants indicated that (HEOR) researchers complied with Open Science practices, such as pre-registering their study, sharing software code and public outreach to a non-scientific audience (Fig. 3).

Fig. 2

Participants’ opinions towards scientific knowledge before and after the teaching innovation. A I trust the results described in scientific journal articles. B All scientific knowledge is produced according to an open and transparent process. C Studies contesting the research hypothesis (negative results) are published as often as studies confirming the research hypothesis. D When a health economic evaluation complies to reporting guidelines, it means that it is highly reproducible. E Researchers are rewarded for performing transparent and reproducible research. F Open-source publishing of health economic evaluations is common practice. G Industry-sponsorship influences the results of health economic evaluations

Fig. 3

Participants’ opinions concerning (HEOR) researchers’ compliance with Open Science principles and practices before and after the teaching innovation. A Researchers pre-register research hypothesis and analysis plan before performing the research in question. B Researchers share data openly. C Researchers share software code used for data analysis. D Researchers take measures to ensure the reproducibility of their research. E Researchers involve stakeholders in scientific knowledge production (i.e. citizen science). F Researchers communicate scientific knowledge to lay audience or perform any type of public outreach

Finally, participants reported less often that they had ‘No knowledge at all’ of Open Science-related skills after the teaching innovation compared with before (except for the skill ‘identifying fake news’). The skills for which at least 50% of participants rated their proficiency level as ‘Advanced’ after the teaching innovation were (1) using a script-based programming language, (2) script annotation and documentation and (3) collaborative writing (Fig. 4).

Fig. 4

Self-reported participants’ level in Open Science-related skills before and after the teaching innovation. A Being able to use a script-based statistical software, e.g. R, Python, … B Pre-registering your analysis plan. C Writing a data and/or software management plan. D Version controlling (script, software, …). E Code/script annotation and documentation. F Licensing software code. G Collaborative (software) script development. H Identifying and reusing open data. I Creating reproducible reports, e.g. using (R) Markdown or Jupiter notebook. J Creating interactive applications, e.g. R-Shiny. K Collaborative writing. L Sharing software code. M Sharing data. N Communicating your work to a lay audience. O Communicating your work through social media. P Critically review someone else’s work and share the review openly. Q Archiving your work in an open online repository. R Adhering to the findable, accessible, interoperable, reusable (FAIR) principles. S Publishing open access. T Identifying fake news

During the group discussions aiming at evaluating the Open Science-related teaching activities, the participants mentioned that the topics addressed during these activities were interesting and relevant for their development as researchers. However, participants mentioned that Open Science encompasses many different concepts and that some of these concepts require knowledge of different digital tools. The practical use of all these digital tools was not yet clear for all participants, most likely because these digital tools were introduced to the participants but they did not need to use them all during the course. Additionally, participants mentioned that they enrolled in this specific course to learn about developing a DES model, rather than for the Open Science-related teaching. Finally, they indicated that the Open Science topics would fit better within the context of a more general course of the programme, such as ‘Research Orientation’ where students learn generic research skills and get acquainted with the different specialisations of the Master’s programme, instead of within a specific simulation course.