Eight characteristics of rigorous multilevel implementation research: a step-by-step guide

Eccles MP, Mittman BS. Welcome to implementation science. Implement Sci. 2006;1:1–3.

Article  PubMed Central  Google Scholar 

Molina-Azorín JF, Pereira-Moliner J, López-Gamero MD, Pertusa-Ortega EM, Tarí JJ. Multilevel research: foundations and opportunities in management. BRQ Bus. Res. Q. 2019; online.

Cleary PD, Gross CP, Zaslavsky AM, Taplin SH. Multilevel interventions: study design and analysis issues. J Natl Cancer Inst Monogr. 2012;44:49–55.

Article  Google Scholar 

Mathieu JE, Chen G. The etiology of the multilevel paradigm in management research. J Manage. 2011;37:610–41.

Google Scholar 

Choi BCK, Pak AWP. Multidisciplinarity, interdisciplinarity and transdisciplinarity in health research, services, education and policy: definitions, objectives, and evidence of effectiveness. Clin Invest Med. 2006;6:351–64.

Google Scholar 

Aguinis H, Gottfredson RK, Culpepper SA. Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling. J Manage. 2013;39:1490–528.

Google Scholar 

Kozlowski SWJ, Klein KJ. A multilevel approach to theory and research in organizations: contextual, temporal, and emergent properties. Multilevel theory, research, and methods in organizations: foundations, extensions, and new directions. San Francisco, CA: Jossey-Bass; 2000. p. 3–90.

Kulik CT. Climbing the higher mountain: the challenges of multilevel, multisource, and longitudinal research designs. Manag Organ Rev. 2011;7:447–60.

Article  Google Scholar 

González-Romá V, Hernández A. Conducting and evaluating multilevel studies: recommendations, resources, and a checklist. Organ Res Methods. 2022;online.

Williams NJ, Preacher KJ, Allison PD, Mandell DS, Marcus SC. Required sample size to detect mediation in 3-level implementation studies. Implement Sci. 2022;17:66.

Article  PubMed  PubMed Central  Google Scholar 

Peugh JL. A practical guide to multilevel modeling. J Sch Psychol. 2010;48:85–112.

Article  PubMed  Google Scholar 

Lane SP, Hennes EP. Power struggles. J Soc Pers Relat. 2018;35:7–31.

Article  Google Scholar 

Paskett E, Thompson B, Ammerman AS, Ortega AN, Marsteller J, Richardson D. Multilevel interventions to address health disparities show promise in improving population health. Health Aff. 2016;35:1429–34.

Article  Google Scholar 

Phillips DC. Philosophy, science and social inquiry: contemporary methodological controversies in social science and related applied fields of research. Pergamon Press; 1987.

Beidas RS, Ahmedani BK, Linn KA, Marcus SC, Johnson C, Maye M, et al. Study protocol for a type III hybrid effectiveness-implementation trial of strategies to implement firearm safety promotion as a universal suicide prevention strategy in pediatric primary care. Implement Sci. 2021;16:89.

Article  PubMed  PubMed Central  Google Scholar 

Kreamer LM, Albritton BH, Tonidandel S, Rogelberg SG. The use and misuse of organizational research methods ‘best practice’ articles. Organ Res Methods. 2023;26:387–408.

Article  Google Scholar 

Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10:1–2.

Article  Google Scholar 

Damschroder, LJ, Reardon, CM, Widerquist, MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implementation Sci. 2022;17:75.

Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Sci. 2019;14:1.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science. 2009;4.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38:4–23.

Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, et al. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:1–1.

Article  Google Scholar 

Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:1–20.

Article  Google Scholar 

Enders CK, Tofighi D. Centering predictor variables in cross-sectional multilevel models: a new look at an old issue. Psychol Methods. 2007;12:121–38.

Article  PubMed  Google Scholar 

Williams NJ. Multilevel mechanisms of implementation strategies in mental health: integrating theory, research, and practice. Adm Policy Ment Health. 2016;43:783–98.

Article  PubMed  PubMed Central  Google Scholar 

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38:65–76.

Article  PubMed  Google Scholar 

Klein KJ, Dansereau F, Hall RJ. Levels issues in theory development, data collection, and analysis. Acad Manage Rev. 1994;19:195–229.

Article  Google Scholar 

Schein EH. Organizational culture and leadership. 4th ed. San Francisco, CA: Jossey-Bass; 2010.

Google Scholar 

Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implement Sci. 2014;9:1–1.

Article  Google Scholar 

Jacobs SR, Weiner BJ, Bunger AC. Context matters: measuring implementation climate among individuals and groups. Implement Sci. 2014;9:46.

Article  PubMed  PubMed Central  Google Scholar 

Kitzinger J. Qualitative research: introducing focus groups. BMJ. 1995;311:299–302.

Article  CAS  PubMed  PubMed Central  Google Scholar 

Zade H, Drouhard M, Chinh B, Gan L, Aragon C. Conceptualizing disagreement in qualitative coding. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. New York, NY, USA: ACM; 2018. p. 1–11.

Headley MG, Plano Clark VL. Multilevel mixed methods research designs: advancing a refined definition. J Mix Methods Res. 2020;14:145–63.

Article  Google Scholar 

Aguinis H, Molina-Azorín JF. Using multilevel modeling and mixed methods to make theoretical progress in microfoundations for strategy research. Strateg Organ. 2015;13:353–64.

Article  Google Scholar 

Hofmann D. Centering decisions in hierarchical linear models: implications for research in organizations. J Manage. 1998;24:623–41.

Google Scholar 

Klein KJ, Kozlowski SWJ. From micro to meso: critical steps in conceptualizing and conducting multilevel research. Organ Res Methods. 2000;3:211–36.

Article  Google Scholar 

Hohmann AA, Shear MK. Community-based intervention research: coping with the “noise” of real life in study design. Am J Psychiatry. 2002;159:201–7.

Article  PubMed  Google Scholar 

Hamilton AB, Finley EP. Qualitative methods in implementation research: an introduction. Psychiatry Res. 2019;280:epub.

Article  Google Scholar 

Finley EP, Huynh AK, Farmer MM, Bean-Mayberry B, Moin T, Oishi SM, et al. Periodic reflections: a method of guided discussions for documenting implementation phenomena. BMC Med Res Methodol. 2018;18:153.

Article  PubMed  PubMed Central  Google Scholar 

Getrich C, Heying S, Willging C, Waitzkin H. An ethnography of clinic “noise” in a community-based, promotora-centered mental health intervention. Soc Sci Med. 2007;65:319–30.

Article  PubMed  Google Scholar 

Lincoln YS, Guba EG. Naturalistic inquiry. Thousand Oaks, CA: Sage; 1985.

Book  Google Scholar 

Patton M. Qualitative research & evaluation methods. 4th ed. Thousand Oaks, CA: Sage Publications, Inc.; 2015.

Google Scholar 

Miles MB, Huberman AM, Saldaña J. Qualitative data analysis: methods Sourcebook. 4th ed. Thousand Oaks, CA: Sage; 2020.

Google Scholar 

Bliese PD. Within-group agreement, non-independence, and reliability: implications for data aggregation and analyses. In: Klein KJ, Kozlowski SWJ, editors. Multilevel theory, research and methods in organizations: foundations, extensions, and new directions. San Francisco, CA: Jossey-Bass; 2000. p. 349–81.

Google Scholar 

Chan D. Multilevel research. In: Leong FTL, Austin JT, editors. The psychology research handbook. 2nd ed. Thousand Oaks, CA: Sage; 2006. p. 401–18.

Google Scholar 

Diez Roux A v. A glossary for multilevel analysis. J Epidemiol Community Health. 2002;56:588–594.

Luke DA, Stamatakis KA. Systems science methods in public health: dynamics, networks, and agents. Annu Rev Public Health. 2012;33:357–76.

Article  PubMed  PubMed Central  Google Scholar 

Comments (0)

No login
gif