To understand the import of the COVID Conversations, the Core Team conducted two small studies. First, we constructed a comparison of the COVID Conversations organizational structure, operations, and activities against criteria for a Learning Community, as derived from related literature. Second, we administered a questionnaire to COVID Conversations participants, asking them to report on their ongoing COVID-related projects and activities.

3.1 Study 1: Characterizing the COVID conversations

The Core Team modeled the COVID Conversations on the organizational structure and characteristics of Learning Health Networks and Learning Communities in order to promote sharing around COVID-19-related activities. However, there were salient differences between this meeting series and the Learning Communities that are formed in clinical and community settings. For example, since DLHS came together rapidly as a community to participate in these COVID Conversations during an emergent situation, there was no time for proactive planning activities in the way a traditional Learning Community or Learning Network might be developed and initiated.5 There was no formal stakeholder identification nor specific health problem of interest designated in advance. Nevertheless, aspects of the COVID Conversations were explicitly modeled on attributes of Learning Communities. Given other attempts to understand the characteristics of these groups,6 we wanted to see if we could meaningfully characterize the COVID Conversations as a variant of a Learning Community.

We retrospectively evaluated the COVID Conversations against criteria that have been developed and published by experts on Learning Health Networks and Learning Communities. While these groups go by different names, they are organizationally similar and draw from similar conceptual foundations. For clarity, we will use the term Learning Communities (LC), but wish to recognize the commonalities between these two ways of achieving the vision of the LHS. Similar to the definitions provided earlier in this paper, Man et al7 describe a LC as providing “a structure for people and organizations to align around a shared purpose and work cooperatively to achieve defined common goals” (p. 227).

Applying general descriptions of LCs, the COVID Conversations appeared, on the surface, to share characteristics with a typical LC. For example, there was a framework in place around which people could come together. The organizational structure of the COVID Conversations was similar to that of other LCs, including having a Core Team and community participants. Also similar to LCs, the COVID Conversations were composed of multiple stakeholders (eg, faculty, staff, students) engaged in projects and activities focused on a common problem.

Upon closer scrutiny, however, we realized that some of the characteristics of the COVID Conversations diverged from those of a typical LC. Primarily, COVID Conversations attendees, although focused on COVID-19 problems in general, were not all working on the same, specific problem of interest. Rather, there was an overarching theme to the work of the members, who were addressing a variety of problems related to COVID-19 with research teams that were often external to, or only partially overlapping with, the COVID Conversations community. This is evidenced by our Study 2 data, which show that when asked about their project goals, participants reported a range of different goals. Thus, while attendees regularly shared updates and received feedback from the group, much of their COVID-related work was occurring with other non-department colleagues or in their personal lives. Curious about other potential differences and similarities between the COVID Conversations and LCs, we compared attributes of the COVID Conversations against criteria of Learning Communities3, 8-11 as described in Table 1.

TABLE 1. COVID conversations compared to LHS learning community characteristics Learning Community characteristic Apply to COVID conversations? Demonstrated by Pursues a shared goal or problem Partial The community came together at an overarching level to address the impacts of COVID, in general. However, there were varied research, educational and personal projects and initiatives related to COVID-19 that were being worked on by individual community members. The community as a whole was not working toward interventions intended to address a specific, shared COVID problem of interest. Driven by “passion” to achieve the goal Yes The passion to help mitigate, understand, and make contributions to the COVID problem was evidenced not only by the quantity and quality of the professional and personal projects and activities undertaken by department members, but also via participation in the weekly community meetings to share progress and update the commons. Not top down; has a leader that is a facilitator Yes The DLHS Department Chair sparked the idea for a weekly meeting series that became the COVID Conversations. The faculty lead of the Core Team acted in support of the community and facilitated the weekly meetings. She also worked with other Core Team members to design and build infrastructure for the COVID Conversations. Multi-stakeholder, collaborative and practical Yes Department members participated in COVID-related activities and were invited to attend weekly COVID Conversations. This group included faculty, staff and students. Shares results and learns from each other Partial While department members shared their COVID-related work in a variety of ways (eg, through the commons, through presentations and updates at regular community meetings), the shared results were not about, or specific to a common problem of interest. COVID Conversations provided an opportunity for group members to share news during the open discussion. Continuous Pending COVID Conversations took place regularly from March-December 2020; it is still to be determined how long the community will persist. Accountable Yes While there were no achievement metrics specified for the COVID Conversations, members were informally accountable to each other in the following ways: Attending meetings regularly Sharing the spotlight by volunteering to report on project progress at weekly meetings Updating the commons with project updates Employs strategies that are “co-produced”; progress comes from the whole community No Since not everyone was working on the same COVID-related project or activity, co-produced strategies targeted for a specific problem was not applicable in this case. Ensures no one dominates Yes The Core Team was motivated to create a welcoming, anti-hierarchical environment where discussions of research were not more important than discussions of teaching, personal activities, or clinical work, and where an intentionally wide variety of speakers were chosen to present their projects at meetings.

Based on this comparison of attributes of the COVID Conversations to those of traditional LCs, we identified three significant differences. The first, mentioned in the previous section, is that the COVID Conversations were not focused on a shared problem of interest. Rather, the projects led by participants had a variety of different goals related to tackling the COVID-19 pandemic. The remaining two differences were that the COVID Conversations did not facilitate the sharing of results and learning from each other around a common problem, nor did members employ strategies or interventions to address a shared problem that was co-produced by the entire community. Neither is surprising given that both of these characteristics are connected to a typical LC, which is formed to identify, scope, and collaboratively address a shared problem of interest.

After analyzing the different facets of the COVID Conversations as compared to LCs, in addition to identifying some key differences, we realized that the COVID Conversations incorporated an element that is not often explicitly mentioned in the literature describing LCs: social support (although see Fagotto12). Not only did the COVID Conversations provide an opportunity for people to come together to talk about their COVID-related work, but they also provided a chance for participants to support each other in the early months of the pandemic. Regular “Wellness Corner” presentations made community care and social support explicit aspects of the COVID Conversations initiative.

Although many COVID-related activities took place outside of the Department of Learning Health Sciences, the pandemic also occasioned new collaborations within DLHS. One such effort was the work involved in producing a DLHS-sponsored webinar in early June, 2020 called “Learning Health Systems in the Time of COVID-19”.13 This webinar, collaboratively developed and delivered by several department faculty and staff, introduced individuals to the overall concept of LHSs with special emphasis on addressing problems created by the COVID-19 pandemic. Another group of collaborators partnered to study the university's efforts to produce face shields in-house during the personal protective equipment supply chain collapse.14 These examples of new collaborations represent the extension of existing collegial relationships in the new context of COVID-19.

In analyzing and identifying key differences between the COVID Conversations and LCs, we recognized a new type of Learning Community. While sharing many of the same attributes of traditional LCs, this new type of community consists of individuals working together to coordinate similar work around a broad problem and, at the same time, provides social support for its members. Based on this analysis, we refer to the COVID Conversations as a “Meta”-Learning Community.

3.2 Study 2: What activities were department members engaged in?

The goal of our second study was to gather additional details about department members' projects and activities. Information about these activities was gathered using a questionnaire. The questionnaire and study protocol were reviewed by the University of Michigan Medical School Institutional Review Board (IRBMED) and determined to be exempt from ongoing review.

We developed a process to understand the nature and scope of the different projects and activities being carried out by department members. The first step was to promote use of the Activity Tracker to document COVID-related activities department members had planned or were working on. Using the Activity Tracker, 28 individuals reported 78 activities. The information collected included the project name, a brief description, and names of team members. Because one person reported the project, both in the activity tracker and in the questionnaire described below, our number of individual respondents remains low despite a relatively high number of activities reported. Next, we designed a taxonomy to categorize research, educational, clinical, and operational activities to understand if and how they related to common Learning Health System attributes and activities. The taxonomy was developed iteratively: a first version was developed based on categories derived from our inductive assessment of the initial project descriptions and components of the LHS infrastructural services model proposed by Friedman et al.10 A compiled set of categories was chosen and presented during a COVID Conversations meeting for review and comment. Based on the feedback received, we adjusted and developed a second version with additional categories. The final taxonomy is presented in Table 2.

TABLE 2. Taxonomy of project characteristics Nature LHS cycle Scale Status Infrastructural services Governance structure Funding type Organizations involved Goal Intended outcome Keywords Research (basic or applied) D2K Local Planned Organize, Start, Maintain, and Support Learning Communities Individual Activity Internal/Departmental DLHS Prevention/Risk Assessment Improve Diagnostic Accuracy Information technology resources; Telehealth; Product design; Bed capacity; Procurement/supply chain; Blood bank; Staffing; Predictive analytics; Public policy; Organizational rules, regulations and policy; Quality and safety/Implementation; Community service; Ethics; Education; Curriculum design Outreach/Dissemination (webinars, podcasts, etc.) K2P Regional Current Capture, Identify, and Measure Performance and Performance Changes Department Venture (DLHS) Internal/U-M THSL Diagnostics/Testing Infection Prevention Design and Development (manufacturing, product/tool design, programming) P2D National Completed Represent Health Information as Computable Data Multi-Department Venture (DLHS + Internal Med) Extramural Michigan Medicine Reducing Disparities in Health Education Restore/maintain operations in educational settings Curriculum/Education Learning Communities Global Terminated/Not Completed Provide and Govern Access to and Use of Data Share and Analyze Data Into New Knowledge Michigan Medicine Venture Special COVID-19 Funding Opportunity MUSIC CQI Treatment-inpatient Facilitate safe staffing models Personal or community activity (advocacy work, making masks, donating blood, etc.) Other Other Share and Analyze Data Into New Knowledge University of Michigan Venture Professional development funds/self-funded MDHHS Treatment-outpatient Sharing knowledge with community Other Other Make Knowledge Computable and Sharable Multi-Organization Joint Venture or Partnership Unfunded Other Clinician communication/coordination Other Generate and Deliver Knowledge-Derived Advice to Applicable Users Other Other Understanding Social Determinants of Health Enable and Promote Performance Changes Dissemination Contribute to Educational Efforts Clinical research Other Education/training Other Note: Table should be read by column only, not by row.

After developing the taxonomy, we decided to test it using a questionnaire (Table 3) and, as a byproduct, gather additional information about COVID Conversations participants' activities. On June 10, 2020, the questionnaire was sent via email to 28 department members who had logged activities in the commons. A subsequent announcement was sent to all COVID Conversations attendees to invite additional members to participate. The questionnaire was active until July 7, 2020. We received responses from 13 individuals, who reported details on 28 projects. Of these, 10 respondents reported details on 24 projects that were characterized as research, educational, clinical, and operational. For the purposes of this study, we excluded the three respondents who reported four personal activities, as these could not be mapped onto the taxonomy we developed. The reports on 24 projects represent 30% of the potential 78 activities logged in the Activity Tracker.

TABLE 3. Questionnaire Introduction questions 1. Which of these best describes the nature of the project/activity you are logging? (forced choice: Research, Outreach/Dissemination, Design and Development, Curriculum/Education, Personal Activity) 2. Is this activity currently represented in the Activity Tracker (google sheet) on the DLHS COVID-19 Canvas site? (forced choice: Yes, No, Not Sure, Other) 3. What is your name? (free response) 4. What is the name of the activity or project you are logging with this form? (free response) Taxonomy questions (for respondents reporting Research, Outreach/Dissemination, Design and Development, or Curriculum/Education activities) See Table YY for answer options; each question corresponds to a column of the taxonomy. For clarity, questions are marked here as forced choice, select all that apply, or free response 5. Where in the LHS lifecycle is this project/activity? (forced choice) 6. How would you describe the scale of your project/activity? (select all that apply) 7. What is the status of your project/activity? (forced choice) 8. What infrastructural services will your project/activity contribute to? (select all that apply) 9. What is the governance structure of your project/activity? (forced choice) 10. What organizations are involved in your project/activity? (Please add any/all using the Other option; select all that apply) 11. What is the project/activity's goal? (select all that apply) 12. If applicable, please state the intended outcome of your project. For example: improving diagnosis accuracy, infection prevention, restoring/maintaining operations in educational settings, facilitating safe staffing models, sharing knowledge with a community, and so on (free response) 13. What types of deliverables will your project/activity produce? (free response) 14. Please indicate the source of funding for your project/activity, (select all that apply) 15. Please select any keywords that describe your project. Please add additional keywords using the “other” option, (select all that apply) Personal activity questions (for respondents reporting personal or community activities) 16. What is your name? (free response) 17. Please tell us about your personal or community activity (free response) COVID conversations participation questions 18. Please help us understand your participation in the DLHS COVID Conversations (forced choice: I have attended every meeting; I have attended regularly, but not every meeting; I have attended meetings occasionally; I have not attended a COVID Conversation; Other) 19. How would you describe the value of the COVID Conversations to you? (free response) 20. Our goal is to make the COVID Conversations helpful for our DLHS community. Please share any feedback or questions you have for us as we continue to plan these meetings (free response)

Finally, questionnaire responses were analyzed and shared at two COVID Conversations meetings to promote a collective understanding of department members' activities. The highlights of this analysis are outlined in the findings section below. Table 4 presents an overview of the key findings related to the work of the department around COVID-19.

TABLE 4. Summary of DLHS COVID projects' characteristics Project characteristics Resultsa % (number of projects) Project status Completed 37.5% (9) Current 50% (12) Planned 12.5% (3) Project funding source Unfundedb 41.7% (10) Extramural 25.0% (6) Internal/Department 20.8% (5) Internal/Department, Extramural 4.2% (1) Professional Development Funds, Self-Funded, Unfunded 4.2% (1) No data 4.2% (1) Project scalec Local 39.1% (18) Regional 23.9% (11) National 21.7% (10) Global 15.2% (7) Number of organizations per project One 58.3% (14) Two 12.5% (3) Three 4.2% (1) Four 4.2% (1) Five 0% Six 4.2% (1) Multiple 8.3% (2) N/A 8.3% (2) Goalsc Clinical Research 5.2% (3) Clinician Communication/Coordination 5.2% (3) Department morale and cohesion 1.7% (1) Diagnostics/Testing 8.6% (5) Dissemination 12.1% (7) Education/Training 12.1% (7) Evaluation 1.7% (1) LHS webinar series 1.7% (1) Organizational 1.7% (1) Perception of telehealth by healthcare providers and patients 1.7% (1) Prevention/Risk Assessment 6.9% (4) Reducing Disparities in Health/Education 13.8% (8) Treatment: Inpatient 6.9% (4) Treatment: Outpatient 8.6% (5) Understanding how learning communities work 1.7% (1) Understanding organizational coordination in crisis response 1.7% (1) Understanding Social Determinants of Health 8.6% (5) a Source: Department questionnaire. b Projects marked as unfunded should be considered to be funded by department resources. c Accounting for some respondents indicating their projects spanning multiple scales and goals.

We identified five key findings from the analysis of questionnaire responses:

Urgency was important. As of July 2020, 37.5% of the COVID project work reported by questionnaire respondents was completed between early April and the end of July. Another 50% of projects were in process and another 12.5% were planned. Due to the unique challenges posed by the COVID-19 pandemic, projects were required to produce results that could be implemented quickly. This finding may have implications for Learning Communities in terms of how and when they form and operate. In the context of COVID-19, problems of interest required urgent actions and quick solutions. Thus, when addressing urgent problems, the formation of a Learning Community may need to take place reactively instead of proactively.

Funding considerations were less crucial. Given the importance of addressing the effects of the COVID-19 pandemic, speed appeared to rise above the need for funding in the execution of COVID-related projects. The upshot was that many activities, particularly near the beginning of the pandemic, were carried out in the absence of dedicated funding, drawing on funded department time and faculty discretionary funds rather than formal grant mechanisms. During COVID Conversations, discussions were focused on projects' progress, and participants were able to connect around common problems they were facing and resources or contacts that could be shared.

Engagement and collaboration occurred locally. In terms of project scale, one dimension of the taxonomy, 39.1% of the COVID-related projects reported were focused on local activities, 23.9% on regional, 21.7% on national, and 15.2% of projects were focused at a global scale. In addition, 58.3% of projects involved one organization and 12.5% involved two organizations, with the remaining projects involving three to more than six organizations (8.3% of respondents noted that this question was not applicable to their efforts). This suggests that projects were focused on relatively local solutions, perhaps because of the organization-specific initiatives that department members were asked to help study and contribute to during the pandemic response.

Projects and activities were focused on the Knowledge to Performance (Practice) (K2P) area of the Learning Cycle. When asked what learning cycle infrastructural services10 the COVID-related activity contributed to, responses varied around the cycle, including answers about activity not explicitly represented on the learning cycle (eg, Education; see Figure 1). However, in total, 40% of the projects were identified as being focused on the Knowledge to Practice (K2P) portion of the cycle. This indicates an emphasis on the implementation of knowledge and performance improvements, including addressing urgent needs faced by patients and clinicians during the pandemic, as opposed to basic science research, which would align with the Data to Knowledge (D2K) part of the cycle. Nonetheless, features of the installed infrastructure for the reported projects could have also been limiting factors that led researchers to focus their efforts on specific parts of the learning cycle. That is, if researchers found that the infrastructural capabilities for P2D and D2K were less developed than those available for K2P activities, researchers may have decided to focus their efforts on the latter to respond most rapidly to the pandemic.

DLHS COVID Project Infrastructure Focus (adapted with permission from Friedman et al10)

Projects had diverse goals. The three most cited goals were Reducing Disparities in Health/Education (14%), Dissemination (12%), and Education/Training (12%), as shown in Table 4. The first category is not surprising, given the broader socio-cultural context, especially protests for racial justice during the summer of 2020 and the consequently articulated need for further research aimed at reducing health disparities. Dissemination, Education, and Training are also intuitive goals of COVID-related activities. Other significant responses included Diagnostics/Testing (9%), Treatment (Outpatient) (9%), Understanding Social Determinants of Health (9%), Treatment (Inpatient; 7%), and Prevention/Risk Assessment (7%). These diverse goals highlight the fact that participants were not working on the same specific problem of interest. Rather, they were addressing a variety of problems related to the complex phenomenon of the COVID-19 pandemic.

There are limitations to the findings presented here. While we did hear from 10 respondents who represented 24 of the 78 of the activities reported by department members (30%), this is overall a modest sample. The descriptive analyses presented here should be interpreted as trends, and not as perfectly representative of department members' activities. However, as discussed above, this analysis did reveal some interesting patterns, particularly the greater number of projects focused on addressing the knowledge to practice (K2P) stage of the learning cycle.

While based on a small sample, we believe this taxonomy develops a framework that may be used to characterize other Learning Communities as instantiations of the LHS model. This could serve several purposes related to understanding the activities and maturity of an LC, and could be used in conjunction with other frameworks, such as the maturity grid designed by Lannon et al.15 Specifically, the taxonomy could aid in self-assessment of the current state of an LC based on the taxonomy categories, including understanding the goals of the LC, the scale on which it is intended to operate, its funding and governance structure, and the organizations involved. Second, it could assist LCs in planning for improvements, such as planned action items for changes in organizational structure, including expanding the scope of the LC's goals, expanding the number of organizations in the LC, and modifying funding sources and governance structure. Third, it could be used for benchmarking with other LCs using the same standard taxonomy, which could lead to important insights or best practices for optimization of their work. Finally, the taxonomy could support new groups in the design phase, particularly in identifying organizational and governance structures based on available knowledge of what has worked elsewhere. We consider the initial development of this taxonomy itself to be an important outcome of our self-study of the COVID Conversations and our concerted conceptual development of the “Meta”-Learning Community.