Study design

We report this protocol using the SPIRIT guidelines for reporting of intervention trials (see Additional file 1) [25]. This protocol uses a type 3 hybrid effectiveness-implementation design, which prioritizes testing the SSL implementation strategy while gathering information on the impact of CM on patient outcomes [26]. We prioritize implementation outcomes, because prior work has established robust support for CM as an effective treatment for stimulant use [27]. We use a hybrid type 3 design that includes measures of clinical effectiveness instead of an implementation trial, because successful implementation does not always result in improved patient outcomes [28, 29].

This trial uses a unidirectional crossover stepped wedge design (i.e., all sites crossover in the same direction from usual care to SSL) with 10 OTPs. A stepped wedge design is a sequential roll-out of an innovation (i.e., the SSL implementation strategy) over several discrete time points or “steps.” Hemming and Taljaard [30] recently argued that stepped wedge trials pose greater risk of misspecification than cluster randomized trials and should only be used when the stepped wedge trial meets at least one of four overlapping conditions are met: (a) provides a means to conduct a randomized evaluation that would not otherwise be feasible; (b) enhances the acceptability of a randomized evaluation to key gatekeepers; (c) is the only feasible design due to practice and logistical considerations; and (d) has increased statistical power over other study designs. This study meets all four of these conditions. We are partnering with the New England ATTC, which developed the SSL and is tasked with working with Single State Authorities to support the addiction treatment and recovery support workforce. The Single State Authorities were not amenable to randomizing OTPs to two different strategies: the Single State Authorities asked that all OTPs receive the same implementation strategy for equity reasons. Our partners at the New England ATTC also preferred to offer all agencies the same implementation strategy consistent with their usual operations.

As shown in Fig. 1, 10 OTPs crossover from implementation as usual to the SSL strategy across four discrete wedges or cohorts. At the start of each wedge, 2–3 OTPs are randomly assigned to crossover. Within each cohort, we apply the Exploration, Preparation, Implementation, Sustainment [31] process model to divide implementation activities into discrete implementation phases. Exploration activities were completed by our team in prior work designing this study [18]: the SSL strategy condition will therefore contain only Preparation, Implementation, and Sustainment activities.

Fig. 1

Stepped wedge design and timeline. Ten opioid treatment programs are randomized to crossover from usual care (depicted in grey) to receipt of the Science to Service Laboratory across four wedges. The Science to Service Laboratory consists of preparation activities (depicted in blue) and active implementation activities (depicted in yellow). Following 6 months of active Science to Service Laboratory support, the programs shift into a sustainment phase (depicted in green) during which they receive minimal support. Programs provide data at the 0, 6, 12-, 18-, 24-, and 30-month marks

During the first wedge, two OTPs will be randomly selected to crossover from usual care into the SSL implementation strategy condition. These two OTPs receive the SSL elements corresponding to the Preparation and Implementation phases of implementation. During the second wedge, the next cohort of two OTPs receives the SSL elements corresponding with the Preparation and Implementation phases and the first cohort shifts to the Sustainment phase, during which SSL activities are stopped and sustainment is tracked. The crossover pattern continues until all OTPs have received active SSL support. Randomization will be done using a random number generator prior to the start of each wedge by the study statistician. Immediately after randomization, he will inform the study team, who will commence scheduling of Preparation activities with the randomly selected sites. To ensure decoupling of financial support with the randomization to the SSL, OTPs receive a flat stipend each year that they participate in the study.

A key benefit of the stepped wedge trial is the extended timeline to measure sustainment. Over a 30-month study, the stepped wedge design allows the first wedge to provide sustainment data for up to 24 months.

Participant recruitment

Reflecting the multi-level SSL strategy, we recruit participants at three levels.

Organization-level: OTPs

We will recruit 10 OTPs in New England serving a mix of urban, rural, and suburban communities, with patients from a broad array of socio-economic and racial/ethnic backgrounds. Eligible OTPs must meet the following criteria: (a) prescribe FDA-approved medication to treat patients with an opioid use disorder and concurrent stimulant use; (b) enroll at least 5 new patients per month; and (c) have at least 3 counselors who offer psychosocial support to OTP patients. We estimate that approximately 8 patients will be observed via electronic medical record (EMR) pulls per month at each OTP. Across 10 OTPs and 30 months of observation, this results in an estimated 2400 EMR pulls, across 70 counselors (assuming counselor turnover).

Research staff will recruit OTPs via established relationships with Departments of Health throughout New England. The Departments of Health host regular meetings with OTPs throughout their state and the research team will be invited to present a study overview at one of these meetings. The study overview consists of a standard set of presentation slides and a visual flyer that summarizes benefits of participation. The research team used this approach to efficiently recruit 28 OTPs in their prior cluster randomized trial.

Provider-level: OTP counselors and leaders

Across the 10 OTPs, all counselors and leaders (e.g., clinical supervisors, program directors) engaged in direct treatment provision or oversight will be invited to participate. Eligible counselors must: (a) provide ongoing psychosocial support to OTP patients (e.g., individual and/or group counseling sessions), and (b) have an active caseload. Eligible leaders must supervise or manage frontline CM counselors.

Once an OTP administrator indicates interest in having their organization participate in the study, the administrator will provide a list of leaders and counselors. Research staff will outreach to counselors and leaders to invite them to enroll via a combination of email, phone, or text, with follow-up visits to the OTPs as needed. Providers and leaders will be invited to receive the SSL support regardless of whether they enroll in the study: enrollment will determine whether they provide evaluation data.

Patient-level: OTP patients

Eligible patients will be newly initiated on MOUD (within past 30 days) at the participating OTPs and have concurrent stimulant use (indicated by self-report of stimulant use or positive toxicology screen within past 30 days). We focus on patients newly initiated on MOUD because need is highest among these patients: drop-out rates and missed doses are higher during the first 6 months in any subsequent period [5, 6, 32]. Data from these patients will be extracted from the electronic medical record.

CM intervention

The CM protocol that will be implemented is Petry’s evidence-based low-cost CM targeting stimulant abstinence [10, 33, 34], which provides immediate reinforcement via prize draws on an escalating schedule. We chose this CM model because of literature supporting better outcomes for patients with stimulant use when: reinforcement is provided immediately [35]; abstinence is the target [36]; and incentive values escalate to promote retention and enhance the patient’s motivation over time [35, 37, 38]. We simplified Petry’s schedule based on formative research with OTP providers who requested simple, weekly draws [13] and a prize system that could be feasibly sourced (i.e., gift cards).

For this study, partner OTPs will administer one random toxicology screen to CM patients per week for 12 weeks. OTPs may choose to test other substances, but reinforcement will be contingent on a negative stimulant result. OTPs may choose which provider (e.g., nurse, counselor) will administer the screen. The CM counselor of record will review the results with the patient.

Patients will earn prize draws for each stimulant negative sample. Draws increase by one for each negative screen for 12 weeks, yielding a maximum of 78 draws for patients who submit 12 negative samples (i.e., 1 + 2 + 3 + 4 + 5 + … + 11 + 12). A refused or missed screen (i.e., unexcused absence on a testing day) resets draws for the next negative sample down to one, with draws again escalating for sustained abstinence. Patients draw prizes from fishbowls containing 500 slips of paper; 250 state “good job!” but are not associated with a prize, 209 state “small,” 40 state “large,” and one “jumbo.” Using these probabilities and magnitudes of $5, $25, and $100 for the three respective prize sizes, each draw has an average cost of $4.21. Thus, for a 12-week protocol, patients maintaining abstinence would be expected to earn an average of 78 draws × $4.21 draw = $328 in prizes. This maximum anticipated earning rate is identical to the rate used by Dr. Petry’s team in the VA rollout [10]. Of note, the average cost of CM in the VA was only $150 per patient, due to missed sessions resulting in resets back to 1 draw [10]. Ensuring OTPs can sustain this investment after removal of support is a key aspect of SSL external facilitation.

The Science to Service Laboratory (SSL)

Once OTPs are randomly selected to crossover from usual care to active implementation, they will receive the SSL strategy. Detailed descriptions of each element of the SSL strategy are provided below and an overview is shown in Fig. 2.

Fig. 2

Elements of the Science to Service Laboratory

Preparation phase

Each wedge begins with 2 months of Preparation activities. At the start of this phase, counselors and leaders will complete the first of three provider surveys and a team-based inventory.

Didactic training

A CM-delivery expert will lead a full-day didactic workshop for each OTP. The first half of the day will consist of didactic instruction in CM principles, review of videotaped exemplar CM sessions, and demonstration of CM sessions reinforcing abstinence. The second half of the day will consist of modeling CM role-plays. During the last hour of the workshop, counselors will break into pairs to complete role plays using standardized patient scenarios. Counselors will take turns playing the patient and the provider. Role plays will cover a range of possible situations in which CM counselors would have to describe CM to a new patient, provide reinforcement to a patient testing negative, and withhold reinforcement from a patient testing positive. At the end of the training, attendees will complete a 20-item CM Knowledge Scale, which requires attendees to understand and apply CM principles to case vignettes.

Performance feedback

The CM Competence Scale [39], a fidelity scale measuring nine key CM session elements and three general CM skills, will be provided during the didactic training along with a rating manual. This scale will be used to guide provision of performance feedback. Items are scored on 7-point Likert scales: an average score of 4.0 out of 7.0 is considered an adequate fidelity score [34].

During the preparation phase, all OTP counselors will receive performance feedback on two practice cases using the CM Competence Scale. First, the role plays at the end of the training workshop will be observed by CM experts on the research team to determine if each counselor demonstrated adequate fidelity. Live performance feedback will be provided immediately after each role play. Counselors who do not exhibit adequate fidelity during the live role-play will be given corrective feedback and the opportunity to complete another role play until each element of the CM Competence Scale is covered. Second, counselors will have 1 month following the workshop to submit an audio recording of a role-played CM session, with another CM counselor or leader at their OTP. Role plays will use a standardized patient script and be rated by blind coders. CM counselors who demonstrate satisfactory competence will be given positive performance feedback whereas those counselors who do not will be given corrective feedback. OTP leaders will be encouraged to complete fidelity ratings of two role-play sessions to gain familiarity with using the CM Competence Scale as a performance feedback tool.

Additional feedback will be provided via a 20-item CM Knowledge test. CM counselors scoring ≤ 16 (75%) on their test will receive corrective feedback. Leaders will receive a group-level summary of CM Knowledge scores by item for their OTP, to identify areas in need of further training or remediation.

External facilitation

External facilitation in the SSL follows the protocols outlined in the Implementation Sustainment Facilitation strategy, which has demonstrated effectiveness in a cluster randomized trial [40]. The SSL facilitator will use the publicly available facilitator workbook, meeting agendas, and worksheets for the Implementation Sustainment Facilitation strategy [41].

Throughout the Preparation phase, the SSL facilitator will hold monthly site-specific meetings to prepare for CM implementation. Topics covered in these meetings will include: discussing the goals of the implementation initiative; encouraging staff to attend training; addressing any emergent issues related to staff engagement in training; and increasing organizational preparedness for CM implementation. In addition, the facilitator will conduct an in-person site visit to engage in strategic planning for CM workflow integration. Data from the scales given at the start of the preparation phase (leadership engagement, implementation climate, CM attitudes) will be used to identify potential barriers and facilitators to CM implementation and to develop a plan to address them. The meeting will also consist of a CM process walkthrough. A major focus will be on ensuring that the necessary resources will be allocated to launch CM.

Implementation phase

When preparation activities are complete, OTPs shift into the active Implementation phase lasting 4 months. The goal of this phase is to commence CM implementation and integrate it into the usual workflow.

Didactic training

The team’s CM expert will offer monthly calls for OTPs. These calls give OTP counselors and leaders the chance to ask questions about CM techniques. Call content will focus on building CM knowledge and awareness and will not be focused on implementation.

Performance feedback

CM fidelity monitoring will be enhanced by the CM Tracking Tool, an Excel tracking tool developed by the study team. The tool prompts counselors to complete a brief Weekly Report for each CM patient. Each report takes 2–3 min to complete and collects date of the CM session; results of the stimulant urine screen; number of CM draws administered; and specific draws received. It also contains the first 6 items of the CM Competence Scale, which counselors complete as a self-reported fidelity check. Data input into the CM Tracking Tool yield a user-friendly dashboard for counselors that shows each patient’s progress through the CM program, summarizes fidelity of CM delivery, and automatically calculates the number of draws earned and number of draws anticipated in the next session. The CM Tracking Tool is used as the basis of a recommended EMR build for the OTPs and serves as a viable low-tech alternative.

External facilitation

In addition to monthly calls with a CM expert, OTP counselors and leaders will have monthly site-specific meetings with an external facilitator, following the protocols from the Implementation Sustainment Facilitation workbook [41]. Facilitation meetings are used to review each OTP’s implementation progress and discuss ways to improve performance. In the final month, the facilitator will lead a sustainment-centered planning retreat to (a) review center performance during the Implementation phase, (b) discuss the extent to which, and how, the center plans to continue CM implementation; and (c) develop a concrete sustainment action plan. Given that inadequate funding is a key barrier to CM sustainment, a primary focus of the facilitator’s work will be helping organizational leadership to evaluate a range of options to financially support CM, including soliciting prize donations, applying for grant funding, and/or applying for state funding.

Sustainment phase (remaining wedges)

The focus of this phase is having OTPs sustain CM without formal support. OTPs will continue to have access to the CM tracking tool and to the monthly all OTP calls with a CM expert. OTPs will also be encouraged to provide ongoing performance feedback to CM counselors as part of their ongoing operations.

Provider turnover plan

Throughout each stage of the SSL, we anticipate that a substantial proportion of counselors and leaders who consent will leave their OTP or the field entirely, and that new providers will come on board. Provider turnover research [42,43,44] estimates an annual turnover rate of approximately 30% among substance use disorder front-line counselors and 20% among OTP leadership. Our more recent experiences with OTPs recorded comparable rates: 32% of counselors turned over within our 6-month active implementation phase.

To address turnover, a key focus of the SSL strategy is on building the capacity of OTP leadership to provide ongoing support. We video- or audio-record SSL training and facilitation activities so leaders have easily transferrable, low-cost training materials, consistent with usual practices at the New England ATTC. Recommended training of replacement providers will consist of watching and/or listening to recordings of SSL activities. Most critically, CM providers will be required to conduct a role play that meets the CM Competence Scale benchmark before being approved to implement CM. Any time a role play is submitted by a replacement provider, a site leader will be encouraged to review it and submit their own CM Competence Scale rating. This will provide ongoing opportunities for leaders to gain practice rating CM sessions for fidelity and to compare ratings to those given by research staff.

Sources of data

Data are collected via multiple methods, including EMR data extraction, provider surveys, and research team observational measures, described in detail below.

EMR extraction and CM tracking tool

OTPs will document CM session delivery in their usual session records within their existing EMR systems and extract a de-identified dataset including all patient encounters at six time points (start and end of each wedge). OTPs will also be given the Excel CM Tracking Tool to facilitate ongoing tracking and provision of real-time feedback, and those OTPs that opt to use the tool will be asked to send the Excel tracker at the same six time points. De-identified data from all available patient records will be identified by each site to allow the study team to calculate key study variables at each OTP including reach, effectiveness (patient abstinence and treatment retention), adoption, implementation, and maintenance.

Provider surveys and team-based inventory

CM providers (i.e., both counselors and leaders) will complete three online surveys to assess contextual determinants at the provider level. The provider surveys will be administered at three timepoints: at the start of each OTP’s Preparation phase, midpoint of their Implementation phase, and endpoint of their Implementation phase. In addition, CM providers at each OTP will be asked to collaboratively complete a team-based inventory of contextual determinants of implementation at the start of the Preparation phase.

Research team observational measures

Research staff will complete observational measures of implementation strategy progress (on an ongoing basis) and counselor fidelity to the CM model (at the end of the didactic workshop). These scales will enable us to examine the concordance between provider-self report and research staff observation.

Study measures

To address Specific Aim 1 (implementation effectiveness), we measure the following implementation outcomes: Reach, Adoption, Implementation, Maintenance, and Equity [45]. To evaluate Specific Aim 2 (patient effectiveness), we measure two patient outcomes in line with the Effectiveness element of RE-AIM: Stimulant Abstinence and Treatment Retention. Finally, in pursuit of Specific Aim 3 (exploratory mediators), we evaluate contextual determinants as putative mediators of implementation outcomes; specifically, we assess implementation strategy progress, inner setting factors, and provider characteristics. Table 1 presents a detailed overview of study measures by Specific Aim.

Table 1 Description of study measures, data source, and timing of measurementData analysis

Across 2400 EMR pulls, 720, 480, and 1200 are expected to occur during Usual Care, Preparation/Implementation, and Sustainment phases, respectively. Preliminary analysis will include bivariate and multivariate analysis using correlations, cross-tabulations and regression models to ascertain data structure, statistical distribution, and study artefacts. Analyses for all Aims will be performed using multilevel structural equation models (MSEM) in Mplus 8.10 [51]. We conducted Monte Carlo simulation power analyses modeling treatment effects and outcome variance nested in OTPs (anticipated ICC: 15%) and/or counselors (anticipated ICC: 10%; see Table 2). There is no expected missing data from EMR pulls, although at OTP and counselor levels, analyses will follow an intent-to-treat approach to dealing with missing data. Across all analyses, we will employ a moderator-first approach to the equity of treatment effects to ensure that group-level differences can be validly interpreted and that we are sensitive to differential treatment effects among patient subgroups. Similarly, we will examine whether significant variance in treatment effects occurs at counselor- or OTP-levels (through exploring the size of random effects variance parameters) before concluding whether treatment effects can be validly interpreted as fixed at the patient-level.

Table 2 Results of Monte Carlo simulation analyses examining power to detect aim 1 and aim 2 outcome effects

Aim 1 hypotheses (effect of SSL on CM Reach and CM Adoption) will be tested by examining the effect of treatment condition at a given center at a given time point, for patients nested in counselors in centers (3-level model). Generalized linear mixed models will be used to evaluate treatment effects on rates of CM Reach, Adoption, and Maintenance.

For Aim 2 analyses, generalized MSEM models will be used to model the effect of SSL implementation. Using data from prior CM trials, we assume a rate increase for Stimulant Abstinence from 25 to 40% following SSL receipt, and a treatment retention increase of 30% from 9 to 12 weeks following SSL.

For Exploratory Aim 3 analyses, MSEM models will be used to assess whether putative provider-level mediators that will be assessed via the provider surveys (e.g., implementation climate, implementation leadership, CM perceptions) are associated with change in implementation outcomes. Exploratory outcome analyses will measure putative mediators as latent factors combining measures of SSL fidelity, inner setting factors, and provider characteristics, to assess the extent to which latent factors serve as mechanisms of action for the main effect of the SSL on implementation outcomes using bootstrapped confidence intervals.

Trial status

No organizations or providers have been officially enrolled yet; recruitment is slated to begin in January 2024.