Developing learners’ competence in performing clinical procedures continues to be a challenge for medical education, with many learners reporting insufficient opportunities to practice simple skills. Many practice opportunities are lost because learners are expected to observe a procedure repeatedly before trying it.1 Brief instructional videos have been shown to improve learning of clinical procedures.2,3 However, the ability of instructional videos to support learning is dependent on learners’ access to such resources, particularly at the bedside.4

Creating and publishing online videos of clinical procedures is a growing part of mainstream academic publishing.5 The means to publish educational videos has expanded in recent years, especially with the many Internet channels now available to educators.6 However, not every channel may be appropriate for educational purposes.4 Some channels seem to be relatively respectable in academic circles, such as MedEdPORTAL7 and the Multimedia Educational Resource for Learning and Online Teaching (MERLOT),8 whereas other channels, including YouTube, have been linked to unprofessional behaviors9 and educational superficiality.10 YouTube’s defenders have also described it as “informative and accurate” and useful “particularly … in combination with other learning materials.”11 Despite the availability of these channels, there has been little investigation of the advantages of using particular publishing systems for medical education.

This article describes our process for selecting a platform for online educational videos. We discuss our experience with this platform and offer recommendations for identifying effective ways to disseminate instructional materials online.

The PocketSnips Project

In 2005, a group of health care professionals in Northern Ontario initiated the PocketSnips Project to create a series of clinical skills videos that were to be published online as “reusable learning objects.”12 These were targeted at medical and nursing students, residents, and clinicians in practice. A multidisciplinary project team identified appropriate topics and developed source materials for making the videos.1 We used actors wherever possible and only used patients where a particular condition (such as a dislocated shoulder) needed to be seen. The videos were mostly shot in close-up and were presented with an overdubbed audio narrative to anonymize the participants as much as possible. All participants featured in the videos gave informed consent for the materials to be published as open-access online resources.

The Laurentian University IRB approved the production of this video series. The production process closely followed medicolegal regulations, in particular Ontario’s Freedom of Information and Protection of Privacy Act rules.13

The videos focused on the kinetic elements of each procedure so as to keep video clip lengths short. We published the videos on a dedicated PocketSnips project Web site14 under a Creative Commons license15 that allowed third parties to freely use and modify the resources. The PocketSnips Web site allowed users to provide feedback on any of the videos posted there. After six months, there had been fewer than 200 views of the videos, very little feedback had been provided, and, as such, we felt that we had not managed to engage with the broad national and international populations of health professional practitioners and trainees that we had anticipated. We also encountered technical challenges in video distribution (particularly problems with insufficient bandwidth and slow playback), even at this low usage level, and decided to look for alternative ways to disseminate the material.

Selecting a Publishing Platform

Disappointed with the PocketSnips site’s failure to realize our goals for distributing the videos, we began evaluating alternative publishing channels in early 2008. A subgroup of PocketSnips project members considered four candidate systems: MedEdPORTAL,7 Health Education Assets Library,16 MERLOT,8 and YouTube. At the time, other video distribution channels such as iTunesU and YouTubeEDU were not available. Despite having certain academic advantages, such as peer-review processes and a specific instructional focus, the education-specific platforms met fewer of our requirements than YouTube did (see Table 1). Some project members expressed concerns about using YouTube, in particular the lack of a disclaimer mechanism to limit the authors’ liability for the content presented (a feature which was provided on the PocketSnips project Web site). We mitigated this fact by adding descriptions to each YouTube video recommending that viewers seek further information from the PocketSnips Web site. There were also concerns as to whether an essentially entertainment-oriented channel was suitable for clinical training materials.

Table 1:

Selection Criteria and Ratings of Four Platforms Considered for Dissemination of PocketSnips Clinical Skills Educational Videos, 2008

Often, projects that involve the public distribution of educational materials raise concerns from faculty, such as reluctance to share materials and desire to obtain academic credit for publishing learning materials.17 Such concerns were less important to those involved in the PocketSnips project than selecting a platform that could engage a larger audience and record users’ feedback on the videos posted there. Additionally, we considered whether similar material was already available on the platform, the effort required to publish materials on the platform, the platform’s usability, and the user-tracking information the platform provided.

In the end, the perceived advantages to the PocketSnips Project that YouTube offered outweighed the disadvantages, and we selected this as our alternative publishing platform. We closely monitored uptake of the videos on this platform through regular review of user comments and viewer counts over the first few months for evaluation and quality assurance purposes.

Using YouTube

Loading a video clip onto YouTube required minimal technical skill and was much simpler than doing so on the original local video-streaming server. YouTube automatically compresses the source material into a streaming format for easy access. This also solved our problem of choosing which video format to use because, prior to this, mobile devices did not share a common video format or player. Once we uploaded the YouTube videos, we annotated them with metadata tags and descriptive links that pointed back to the original videos on the PocketSnips project Web site.14 YouTube provides basic feedback tools, including the ability for viewers to add comments and vote on whether they liked a video, and recommender systems (similar to Amazon’s “Customers … also bought” feature). Comments and voting could be disabled for each video at the author’s discretion. Google Analytics were added to YouTube shortly after we created the channel,18 extending the available information about how the videos were being used.

We uploaded the PocketSnips videos (with Creative Commons licenses) in July 2008 to a project-specific YouTube channel called CliniSnips.19 We named the channel by mutual agreement with other groups creating similar video material to create a neutral venue to be shared by all contributors.

The 15 PocketSnips videos we uploaded to YouTube varied between 1:29 and 5:03 minutes in length (mean 2:41 minutes). The titles were Central Line, Circulation Test, Digital Block, Foley Catheter, Intravenous Line, Local Anesthetic, Nasogastric Tube, Shave Biopsy, Shoulder Dislocation, Simple Suturing, Sterile Gloves, Ulnar Gutter Cast, Vandenbos Procedure, Venipuncture, and Volar Slab Cast.

Using simple analytics reports

Thirty-three months after we posted the clips on YouTube, we compiled the data for the 10 most-viewed videos for analysis (see Table 2). Other videos subsequently added to the CliniSnips channel were omitted from detailed analysis. The data came from two sources: analytics data from YouTube (numbers of views, etc.) and the user-generated comments for each video.

Table 2:

Google Analytics-Driven User Data for the Top 10 PocketSnips Clinical Skills Educational Videos on YouTube as of June 18, 2011

YouTube provided analytics data via a standard set of Web-based reports based on Google’s standard analytics suite of tools (note that YouTube’s analytics tool set is now called Insight). We filtered the analytics reports on the basis of date ranges and report functions (e.g., “Demographics”). Some of the calculated reports that YouTube provides are basic and unambiguous, such as the total number of views per video, individual view duration, or viewers’ location; some are more complex and less well defined (such as “Attention,” “Engagement,” and “Discovery”). We collated viewer comments and analyzed them for common themes using an open coding process.

The underlying analytics data are now directly accessible via the YouTube application programming interface (API),20 but we have not had the programming resources to support this approach. Thus, our investigation of the user data was based on the standard analytics reports provided through YouTube. However, Google and YouTube (as its subsidiary) do not publish the algorithms on which their analytics reports are based, so we cannot comment on the underlying methodology used to generate them.

We extracted the analytics data from YouTube in April 2011. There had been more than 1.74 million views on the channel during 33 months, with the most popular video, Intravenous Line, garnering 456,244 views. The mean number of views per video was 163,316. The analytics reports provided a detailed breakdown of viewer behaviors, including the exact amount of time viewers spent watching each video. The total aggregate viewing time for the top 10 videos was more than 74,000 hours. The CliniSnips channel attracted over 2,400 views per day during the 18 months prior to preparing this article. It is particularly encouraging to see on subsequent monitoring that the usage rate has dropped very little, suggesting that the channel has reasonable sustainability. This contrasts strongly with general viewing patterns for YouTube videos, which show a marked decline over about six weeks.21

Despite their simplified approach, the Google Analytics reports we retrieved contained a level of detail that provided more useful information than hit counts alone, such as whether a video was viewed on a mobile device. The proportion of mobile viewers, initially small at around 3%, rose steadily to 9%.

The Demographics report revealed that viewers were 56% male with a modal age of 45. Because of the nature of the videos, we anticipated that they would be most useful for trainees or less experienced clinicians, so the modal age was higher than expected.

The Attention score reflected the time viewers spent on watching each video and is described by YouTube as an “indicator of your video’s ability to hold viewers’ attention compared to videos of similar length.”22 The CliniSnips videos scored highly, with an average of 4.33 ± 0.40, compared with the general mean of 3.3 (out of 5).

The report on Viewing Habits identified those segments of the videos that generated most interest, or were most often reviewed or replayed. Most viewers watched to the end of the clips, with some subtle peaks in the curve where details in the procedural techniques were reviewed several times. We used this particular type of analysis to refine certain clips to better respond to the users’ attention span, including cutting less important segments or annotating segments that had high viewing density to further explain the events being portrayed.

A video’s Community Engagement report was based on the quantity of discussion on the video and the number of links that users made from it to other sites. No details are provided by Google as to how this score is generated, and this lack of transparency made it less useful than other metrics.

The Discovery report reflected how viewers reached a particular video, including whether they arrived from another YouTube video or Web site. Again, the algorithms used are a Google trade secret, but this report view did allow for further exploration as to why there had been an uptick in activity for a particular video. For example, an increase in viewer rates for the Intravenous Line video around January 2011 was correlated with the video featuring on the Medifix video referral site at that time.23 Google Analytics18 also tracked a small but nonsignificant increase in traffic to the original PocketSnips Web site following the videos’ being posted on YouTube.

Social data analysis

YouTube provides viewers with various ways of giving feedback, including approval ratings and text comments. These could be enabled, disabled, or hidden on a per-video basis. Comments and ratings were enabled for CliniSnips content (see Table 2). We compiled the 761 comments from the 10 most-viewed videos and entered them into Atlas.ti (Atlas.ti Scientific Software Development GmbH, Berlin, Germany) for thematic analysis using grounded theory techniques.24 We grouped similar concepts into two main categories to create a common axial coding model: responses, including applause, complaint, critique, discussion, emotional, and flippant responses (examples of each of these are given in Table 3); and educational activities constructed around the CliniSnips videos. The latter had three subordinate codes: (1) health-care-provider-to-patient activities in which, for example, discussion might start with a lay question that would generate responses that appeared to come from health care providers, (2) health-care-provider-to-health-care-provider activities exemplified by discussions between health care professionals about the merits of alternative methods for a particular procedure, and (3) patient-to-patient education activities, such as comments from people with anxiety about an upcoming similar procedure or from people who had already had the procedure, often reassuring the anxious. Although they seemed credible and illustrated some interesting social processes, there was no practical way to confirm that genuine patients or health care professionals made these comments.

Table 3:

Characteristics of User Comments Derived From a Thematic Textual Analysis of Comments on the Top 10 PocketSnips Clinical Skills Educational Videos on YouTube, 2011

Was It Successful?

Our team considered the adoption of YouTube a success because of the huge increase in uptake and the value added by the analytics and comments. The increased exposure of our material also generated additional funding and new opportunities to collaborate with other teams. Many of the team’s original concerns about YouTube, in particular that it was too frivolous for the subject matter, proved unfounded. The modal viewing age was higher than expected for our target audience, although, because the available data are self-declared, the real numbers remain unknown. Given the number of comments originating from what appeared to be patients, this may in part explain the greater modal age. YouTube applied age restrictions for some videos—this appears to be a somewhat arbitrary process, and YouTube does not provide authors notification when this occurs. Further analysis did not reveal that the age restrictions had much effect on the quality of the comments received.

The main drawbacks in using YouTube were its generic page format, an inability to track and identify specific users, and a lack of information on how analytics were generated. We considered that simplified data on very large numbers were much more useful than greater detail on the very low numbers of viewers from our original PocketSnips site. Viewers were anonymous, and we had little indication of whether they represented the target population of health professionals, other than any feedback they left. Specific educational outcomes were therefore difficult to assess in the original target population. Although learning analytics is a growing aspect of using educational technologies,25,26 and the YouTube analytics tools provided many insights into how the videos were used, the analytics’ algorithms are not disclosed by Google and must therefore be considered with significant caution. It is now possible to gain more direct access to YouTube analytics data through their API. This has been useful for other groups, but this was beyond our programming expertise.21,27

In creating a shared CliniSnips channel, we hoped that other video providers would use the channel to share their clinical educational materials. This hope was not realized, however, and CliniSnips has so far only been used for PocketSnips material, with other groups running their own channels. However, the lack of a single portal has been offset by the greatly increased discoverability provided by YouTube through their metadata and search tools, along with the cross-referencing multiplier effect of their automatic recommender system. Examining the referral sources through the YouTube Discovery report has shown this to be a major contributor to the uptake of our videos.

Although user feedback was useful, less than 1 in 2,300 viewers added comments. This low density of contributors is comparable to other reports.27 The level of discussion in the comments was variable and included much trivial content, although it was interspersed with more focused comments that raised a number of useful suggestions. For instance, one discussion thread explored variations on local aseptic techniques. When user feedback did merit a change to the material, as was the case with the Nasogastric Tube Insertion video, revising the sound track to address issues was relatively easy to achieve because we had avoided “talking head” shots when developing the videos. Ultimately, we made eight changes based on the feedback provided during the lifetime of the project. New features in YouTube, added since our detailed analysis period, have enabled more powerful annotation techniques, with cross-linking to other YouTube videos and channels. We are exploring new approaches that allow us to embed links to specific YouTube video segments from our PocketSnips Web site, allowing for more detailed analysis of users’ navigation behaviors.

The authority of the material was not raised as an issue in the comments from viewers, which seems consistent with the findings of other authors.28,29 Quality assurance frameworks for health information, such as DISCERN30 and HONcode,31 raised no issues in an internal review. These frameworks were not well aligned to our needs and mostly focus on information for patients. Our principal focus was on providing information to health care professionals, who less often refer to formal portals and tend to use “foraging” behaviors to find suitable information.29

We did not pursue third-party continuing medical education (CME) accreditation of CliniSnips, and we acknowledge that there were drawbacks to this decision. For instance, of the 74,000 viewer hours logged, at least some of them were likely to have been relevant to CME, but such activity was not eligible for CME credits. Future activity will explore whether CME credits and YouTube publishing can be meaningfully combined.

Our use of YouTube was not an exclusive option, and we continue to maintain the PocketSnips project Web site in addition to the CliniSnips YouTube channel. Although we did not originally select MedEdPORTAL and MERLOT, we are considering exploring publication of the PocketSnips videos on these platforms in addition to YouTube. Publishing the materials through a peer-reviewed model may facilitate the use of PocketSnips videos as registered CME activities.

Recommendations

Our experiences allow us to make several recommendations for others considering different platforms for the distribution of educational materials. First, different channels suit different audiences. Developers of educational materials should have clear goals and align their choice of platforms accordingly. Second, using one channel does not necessarily exclude using others (licensing allowing) to reach different audiences. Third, online technology should be considered to be transitory. Developers of materials should be prepared to move their collections as publishing options change. Finally, analytics tools provide valuable information on patterns of user access and behaviors. Analytics can provide a clearer picture of users’ actions than relying on their opinions and comments.

Conclusions

Our experience offers an innovative approach to disseminating open-access clinical skills training videos. YouTube solved a number of our technical challenges in video publication. It secured a very large audience for the project and, by providing rich data on how the material was used, allowed the project team members to continue to refine the materials to meet the needs of users. Although the density of social interaction was low on YouTube, the absolute quantity was much greater than through our initial project Web site. The levels of engagement we observed on YouTube, in terms of the numbers of views and the discussions, indicate that at least some viewers were using the materials in ways that we anticipated. The modest increase in traffic on the PocketSnips Web site also suggests that some users were willing to take the extra step of exploring the additional materials attached to the videos. Viewers could not be identified as representing a particular professional or educational group. The broader influence of our videos being seen by members of the public has not been assessed, but the effect of short videos on learning is generating much interest. The Khan Academy,32 for example, has had success providing open-access instructional videos at different educational levels on a very wide range of topics.

Our selection of YouTube over more respected publishing channels raised many interesting issues. It moved the project significantly beyond the “read-only” culture of higher education33 by engaging social networking features, such as user comments and ratings of videos. But, by selecting a platform that did not offer peer review, CME credits, or other medical education infrastructure, we were unable to account for the project’s specific medical educational outcomes.

Our use of YouTube evolved over time, and the general acceptance of cloud-based services has increased both within our team and in information services generally. In addition, the way the resources are framed has also changed. For instance, reusability of educational materials used to be the focus in the debate over “reusable learning objects.”12,34 Now, the focus has shifted to access—for instance, in the development of “open educational resources”33,35 and in exploring intellectual property rights. Initiatives such as OpenCourseWare36 and the development of open educational resources, as well as wider awareness of the social responsibility of medical schools, suggest that the experiences reported here have applicability beyond the specific technologies we used. Finally, the use of “crowdsourcing” (i.e., using large numbers of relatively anonymous reviewers rather than a few experts) for reviewing educational materials is a technique that needs further exploration.37,38

In summary, the power of platforms such as YouTube to expose vast audiences to educational resources and then to engage viewers in enhancing them through the use of social media and learning analytics tools affords both advantages and disadvantages. We must continue to study the effectiveness and outcomes of using different publishing platforms for medical educational purposes, both to support the needs of the many stakeholders involved and to support ongoing innovation in using the Internet to advance medical education as a whole.

Acknowledgments: The authors would like to acknowledge the PocketSnips Project team: Andrea Smith, Brian Hart, Sherry Drysdale, Dr. Christine Kupsh, Dr. Laura Piccinin, Dr. Lorraine Carter, Richard Witham, Christina McMillan-Boyles, and Joel Seguin.

Funding/Support: The development of the PocketSnips videos were funded by the Northern Ontario Heritage Fund (www.mndmf.gov.on.ca/nohfc/), the Inukshuk Fund (www.inukshuk.ca), and an unrestricted educational grant from Pfizer Canada (www.pfizer.ca). None of the funders had any influence on content or process.

Other disclosures: The PocketSnips videos are provided free of charge, and the authors receive no income or other material benefits from their development or dissemination. The authors do not endorse the use of YouTube or any other media platform.

Ethical approval: IRB approval was received from Laurentian University for PocketSnips content creation but was not required for the analysis of publicly available data.

References 1. Topps D, Helmer J, Carter L, Ellaway R. PocketSnips: Health education, technology, and teamwork. J Distance Educ. 2009;23:147–156 2. Lee JC, Boyd R, Stuart P. Randomized controlled trial of an instructional DVD for clinical skills teaching. Emerg Med Australas. 2007;19:241–245 3. Gertler S, Ahrens K, Klausner JD. Increased knowledge of safe and appropriate penicillin injection after viewing brief instructional video titled “how to inject Bicillin LA.” Sex Transm Dis. 2009;36:147–148 4. Dinscore A, Andres A. Surgical videos online: A survey of prominent sources and future trends. Med Ref Serv Q. 2010;29:10–27 5. McMahon GT, Ingelfinger JR, Campion EW. Videos in clinical medicine—A new journal feature. N Engl J Med. 2006;354:1635 6. Ruiz JG, Mintzer MJ, Leipzig RM. The impact of e-learning in medical education. Acad Med. 2006;81:207–212 7. Association of American Medical Colleges. MedEdPORTAL. https://www.mededportal.org/. Accessed October 29, 2012 8. California State University. MERLOT—Multimedia Educational Resource for Learning and Online Teaching. http://www.merlot.org/merlot/index.htm. Accessed October 29, 2012 9. Farnan JM, Paro JA, Higa JT, Reddy ST, Humphrey HJ, Arora VM. Commentary: The relationship status of digital media and professionalism: It’s complicated. Acad Med. 2009;84:1479–1481 10. McGee JB, Begg M. What medical educators need to know about “Web 2.0.” Med Teach. 2008;30:164–169 11. Wood A, Struthers K, Herrington S. Modern pathology teaching and the Internet. Med Teach. 2009;31:187 12. Wiley DA The Instructional Use of Learning Objects. 2002 Bloomington, Ind Agency for Instructional Technology, Association for Educational Communications and Technology 13. . Freedom of Information and Protection of Privacy Act. RSO, Chapter F.31. (1990) http://www.e-laws.gov.on.ca/html/statutes/english/elaws_statutes_90f31_e.htm. Accessed November 2, 2012 14. . PocketSnips. Project overview. http://pocketsnips.org. Accessed October 30, 2012 15. Creative Commons.. About the licenses: Attribution-noncommercial-sharealike [in Portuguese]. http://creativecommons.org/licenses/. Accessed October 30, 2012 16. Health Education Assets Library http://www.healcentral.org/. Accessed October 30, 2012 17. Uijtdehaage SH, Contini J, Candler CS, Dennis SE. Sharing digital teaching resources: Breaking down barriers by addressing the concerns of faculty members. Acad Med. 2003;78:286–294 18. Google Analytics. http://www.google.com/analytics/. Accessed October 30, 2012 19. . CliniSnips [YouTube channel]. http://www.youtube.com/user/clinisnips. Accessed October 30, 2012 20. Google Developers. YouTube Analytics API. https://developers.google.com/youtube/analytics/. Accessed October 30, 2012 21. Gill P, Arlitt M, Li Z, Mahanti A. YouTube traffic characterization: A view from the edge. Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement. 2007 New York, NY Association for Computing Machinery:15–28 23. . Medifix. www.medifix.org. Accessed April 23, 2010 [no longer available]. 24. Strauss A, Corbin JDenzin NK, Lincoln YS eds. Grounded theory methodology. Strategies of Qualitative Inquiry. 1998 London, UK Sage Publications:273–285 25. Ali L, Hatala M, Gasevic D, Jovanovic J. A qualitative evaluation of evolution of a learning analytics tool. Comput Educ. 2012;58:470–489 26. van Barneveld A, Arnold KE, Campbell JP. Analytics in Higher Education: Establishing a Common Language. http://www.educause.edu/ir/library/pdf/ELI3026.pdf. Accessed October 30, 2012 27. Cheng X, Dale C, Liu J. Statistics and social network of YouTube videos. In: Proceedings of the 16th International Workshop on Quality of Service. 2008 Twente, The Netherlands:229–238 28. Järvelin K. Explaining user performance in information retrieval: Challenges to IR evaluation. Adv Inf Retr Theory. 2009;5766:289–296 29. Dwairy M, Dowell AC, Stahl JC. The application of foraging theory to the information searching behaviour of general practitioners. BMC Fam Pract. 2011;12:90 30. DISCERN: Quality Criteria for Consumer Health Information http://www.discern.org.uk/. Accessed October 30, 2012 31. Health on the Net Foundation.. HONcode. http://www.hon.ch/. Accessed October 30, 2012 32. . Khan Academy. http://www.khanacademy.org/. Accessed October 30, 2012 33. Wiley DIiyoshi T, Kumar MSV eds. 2005–2012: The OpenCourseWars. Opening Up Education: The Collective Advancement of Education Through Open Technology, Open Content, and Open Knowledge. 2008 Cambridge, Mass Massachusetts Institute of Technology:245–259 34. Littlejohn A Reusing Online Resources: A Sustainable Approach to E-learning. 2003 London, UK Routledge 35. Atkins DE, Brown JS, Hammond AL. A Review of the Open Educational Resources (OER) Movement: Achievements, Challenges, and New Opportunities. http://www.hewlett.org/uploads/files/ReviewoftheOERMovement.pdf. Accessed October 30, 2012 36. . OpenCourseWare Consortium Web site. http://ocwconsortium.org/. Accessed October 30, 2012 37. Surowiecki J The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations. 2005 New York, NY Anchor Books 38. Keen A The Cult of the Amateur. 2007 Boston, Mass Nicholas Brealey Publishing