Consistent with VA’s strategic goal to provide telehealth in the home, OCC began MVAI development in 2015, engaging multiple specialty clinicians, including dermatology, and individuals from information technology (IT), clinical informatics, business, and telehealth leadership. Upon successful pilot testing for technical functionality between May and June 2019 at 3 pilot sites, MVAI was additionally released to groups of 7 new facilities every 3 months beginning July 1, 2019, for a total of 4 groups and 28 facilities.11 Release constituted the opportunity to use MVAI, but OCC did not require facilities to do so and no incentives were offered other than the apps’ intrinsic merits.

To alert and engage key stakeholders, OCC established multiple mechanisms to introduce and monitor MVAI. Two weeks before official release at each facility, OCC sent a brief email to inform the dermatology chief. On the day of release, OCC sent official emails to facility leaders: the dermatology chief, telehealth coordinator, chief of staff, and director, as well as the regional network telehealth coordinator and any other known facility teledermatology champions dedicated to advocating for MVAI. The email provided (1) introductory information; (2) a checklist of tasks aligned with VA’s informatics policy and practice that were necessary to enable interface with the VA EHR; (3) electronic copies of promotional/instructional materials for clinician and patient end-users; (4) links to online videos demonstrating the workflow; and (5) invitations to live online OCC training webinars and biweekly check-in sessions. At least one key individual from each of the exposed facilities attended a webinar, confirming receipt of the emails by each facility. Field notes from webinars and responses to the emails captured a general sense of excitement among participants to implement MVAI, though concerns and skepticism were occasionally expressed.

Clinical champions, who by definition were interested in and dedicated to advocating for change,20 facilitated implementation of policies and practice locally. Champions were largely dermatologists or telehealth leads who self-selected to help complete the set-up process, encourage others to try MVAI, and create systems to provide marketing/training materials to patients. Of the 19 facilities completing at least one bi-monthly report (a 61% response rate), 6 documented that support from clinical champions was instrumental for implementation. Field notes also captured the importance of clinical champions; at one facility, lack of senior leadership support almost led to program discontinuance but for the champion stepping in.

Anticipating the need to capture clinical workload and to facilitate tracking MVAI usage according to VA’s usual practice, OCC created traditional clinical informatics tools and new mechanisms in VCM and other databases to monitor key milestones for MVAI encounters, which were distinct from informatics processes and required separate user workflows.

Reception of these policies and practices were mixed. Once exposed to MVAI, interviewed clinicians found it user-friendly and the workflow straightforward, “It was easy to use…easy to initiate,” “nice to email securely with patients….” Clinicians appreciated MVAI’s intended use cases for both short-term (e.g., “to see how an incision is healing 1-3 weeks later”) and longer-term follow-ups (e.g., for patients with rashes and on biologics). They also found it beneficial not just as a substitute for an in-person visit but to supplement a separate VA-developed app (VA Video Connect or VVC), mediating home video visits. In such cases, MVAI contributed “clearer images for diagnoses.” When clinicians mentioned image quality, most stated that patient-submitted photos “were very good.” Nevertheless, one facility reported image quality concerns, preferring VA’s standard consultative teledermatology utilizing in-clinic imaging.

MVAI’s incomplete integration with VA’s pre-existing EHR, such as the inability to upload images directly to the EHR, concerned some clinicians. For example, VCM enabled clinicians and staff to track MVAI cases but required an extra step outside of the usual EHR workflow. Clinicians’ frustration that patients often did not respond to MVAI requests was an additional barrier identified in all qualitative sources.

All facilities experienced some technical difficulties with MVAI. Uncoordinated system upgrades to VA’s EHR and mobile health infrastructure, which were each overseen by separate VA offices, created brief user problems. Idiosyncrasies with local IT infrastructure ultimately led two facilities to discontinue MVAI use.

MVAI was introduced at facilities that were specifically chosen to have expertise in teledermatology and telehealth in general (see the “Methods” section). Thus, the majority of administrative and clinical support staff, involved in preparing the EHR for MVAI, reported in bi-monthly reports no concerns with their involvement or resources available. Interestingly, the fraction of staff with concerns was greater at facilities introduced to MVAI after the onset of the COVID-19 pandemic (40%) than those exposed pre-pandemic (≤33%). Some concerns arose from misunderstandings, such as when one facility delayed MVAI implementation because it believed MVAI was not an approved protocol during the pandemic.

Patient eligibility was a significant factor in determining climate. Since additional clinic time was not allocated to screen for patient eligibility, use VCM, or educate patients, interviewed clinicians viewed MVAI as a disruption. It was “one more thing… More work without taking away anything.” Highlighting that only ~50% of patients were already enrolled in VA’s mobile health program and that not every patient owned or was proficient with a mobile device, one interviewed clinician stated that it was difficult to “find the ideal patient.” To address these concerns, some facilities used administrative data to pre-identify eligible patients or utilized support staff to assist dermatology clinicians with VCM by creating MVAI requests and to assist patients with MVAI.

The COVID-19 pandemic significantly affected implementation climate. For example, the timing of facilities’ exposure to the app correlated with the rate of achieving implementation milestones (Figure 3). Nearly twice as many facilities introduced to the app pre-COVID achieved each milestone relative to post-COVID facilities, and differences in sustaining use were noticeable. Multiple reasons appeared to explain this phenomenon.

Implementation tracking for MVAI. Facilities were grouped into those exposed to MVAI 6 months or more prior to the onset of the COVID-19 pandemic (group 1); 3 months prior (group 2); at the start of the pandemic (group 3); and 2 months after (group 4). Key milestones for facilities were documented. Sustainment was defined as a facility having a minimum of 2 patients complete a visit and demonstrable evidence of app usage at the end of the study.

First, apart from the time-consuming demands of identifying patients noted above, inadequate staffing itself was a limitation that the pandemic magnified. Nine facilities reported staff shortages, contributing to delays or stoppage in MVAI use. Field notes captured that a facility attempted to implement MVAI but loss of one dermatologist precluded its use. Qualitative data from all sources revealed some concern about the lack of on-site staffing to assist patients. In-person clinic closures due to COVID-19 additionally burdened staff who spent extra time sending brochures/instructions to patients instead of giving them in-person. The common use of part-time physicians also made it difficult for facilities to train all clinicians together with the new workflow (“All seven of us doctors are part time…hard to find time to train”).

Second, competing demands existed. Qualitative data sources from all facilities captured telehealth staff narratives about being tasked with setting up other programs in response to COVID-19, increasing demands on their time and delaying preparations. Facilities also reported either increasing their consultative teledermatology program capacity and/or trying other remote options such as VVC or encrypted emails to receive photos directly from patients at home, both of which were strongly promoted by VA during these years. One Chief of Medicine said using “the app would take away from … clinical obligations.” All qualitative data sources indicated that clinicians experienced increased demands or reassignments associated with COVID-19. One clinician stated, “I can’t run two new programs at the same time.” Clinicians had less time to train for the new workflow.

Attitudes of stakeholders are an important factor in determining implementation climate. Concerns regarding support and resources for the mobile apps from dermatology chiefs, nurses, and facility leadership decreased after COVID-19’s onset, though differences among groups existed. Of 14 facilities responding to a question on the bi-monthly report about whether dermatology clinicians had concerns, 4 (29%) identified major concerns in at least one, though this response did not correlate with when a facility was exposed to MVAI. In contrast, at 4 of 12 facilities responding to a question on the bi-monthly about whether senior leadership had concerns, 3 were among the groups exposed pre-pandemic. Senior facility leadership’s engagement, often important in funding continued operations, is instrumental in creating a strong implementation climate.21 However, using reporting as an indicator of engagement, we found no association between reporting to leadership and use of MVAI at their facility.

Possibly reflecting VA’s strong mission to serve Veterans, bi-monthly reports and interviews elicited that dermatology clinicians were primarily interested in adopting MVAI to improve access, to increase patient convenience, and to make in-person appointments available. They felt MVAI would particularly benefit patients who lived far from a dermatology clinic or who did not want to wait for the next in-person visit.

COVID-19 influenced facilities’ interest in MVAI. Many dermatology clinicians were “committed to implementing whatever telehealth resources were available.” MVAI enabled clinicians to “manage simple patient concerns” remotely to address “the growing backlog of patients” due to appointment cancellations.

During the 17-month study period, 10 (32%) facilities used MVAI to care for 466 unique patients with a total 474 encounters. Nineteen of 31 facilities (61%) implemented MVAI by setting up clinics according to CDW data: 10 of the 19 facilities used MVAI for actual patient care by completing encounters in the EHR; 9 tested it only, reflected by VA mobile health data activity but no formal EHR encounters. Two hundred twenty of 474 (46%) encounters occurred at 7 facilities using MVAI as originally intended with patient-submitted images and history. Six facilities generated MVAI teledermatology encounters technically correctly but deviated by requesting patient-submitted photos without history (9% of all visits)—one facility reported that they adapted MVAI to augment video visits with VVC, suggesting others did as well. Testing accounted for 32% of usage, reflecting intent to implement, while the remaining MVAI encounters (13%) reflected various errors in app usage (Figure 4A).

MVAI usage. Actual MVAI usage reflected by CDW and VA Mobile Health encounter data. Manual inspection was used to categorize encounters by A) test patients (unshaded), and actual patients (shaded areas), and B) low- and high-end clinician user types.

Based on CDW data, use varied by clinician type. An adaptive strategy used by some facilities was to encourage use by residents, nurse practitioners, and physician assistants (NP/PAs), with NP/PAs using MVAI more frequently than attendings. Most MVAI clinicians were resident/fellow trainees, followed by dermatology attendings (Figure 4B). Consistent with these results, two VA facilities that were academic teaching affiliates contributed 84.4% of all MVAI encounters. However, the additional time required to train residents about MVAI was also identified as a burden by 4 facilities in bi-monthly reports.

Although not universal, clinicians increasingly used MVAI as in-person clinics closed due to COVID-19 (Figure 5). To understand the variability in use, we examined self-reported implementation progress, and found that facilities with lower initial implementation stages were associated with the onset of COVID-19 (Figure 6). Of 8 facilities that provided at least one bi-monthly report after the pandemic started, 4 reduced their implementation completion stage, 2 stayed the same, and 2 increased their stage.

Time-dependence of MVAI usage. MVAI piloting was followed by rollout according to a cluster-randomized stepped-wedge design over 17 months. Bars represent the number of facilities recording any MVAI requests for real patients with actual number of facilities stated above each bar. The solid line represents actual number of MVAI requests ordered over all exposed sites. Arrows denote when groups of facilities were exposed to MVAI, starting with pilot sites in May 2019. Asterisk denotes the shelter-in-place orders at the start of COVID-19 pandemic.

Stages of implementation completion (SIC). Bi-monthly reports were used to assess SIC scores for each group (Grp) or cohort of facilities throughout the study period. Mean scores and standard deviations are plotted for each group for each reporting period following exposure to MVAI. Note that because each group was surveyed only after MVAI exposure, Group 1 facilities had the opportunity to complete up to 6 bi-monthly reports, whereas subsequent groups had progressively fewer opportunities. A higher score denotes greater implementation completion.

Of the 10 facilities that used MVAI with patients, 7 sustained usage through the end of the study period, with 3 stopping due to technological barriers or leadership prioritizing other goals.