Printed from: http://www.stfm.org/NewsJournals/FamilyMedicine/PastIssues/ArticleView
FMHubTOC

Using Telemedicine Technology to Assess Physician Outpatient Teaching

Nyann Biery, MS; William Bond, MD, MS; Amy B. Smith, PhD; Matthew LeClair, MA; Elissa Foster, PhD

Background and Objectives: Video conferencing technology (telemedicine) can be applied to many settings within the medical community; we assessed the feasibility of its use in conducting observations of faculty at remote family medicine teaching sites.

Methods: We deployed seven telemedicine units to five family medicine residency sites and two observation stations within our division. Practice managers and physician faculty members received on-site training on the basic functionality of the technology, as well as “best practices” and minor troubleshooting techniques. Quick reference guides and other support documents were developed and provided for each site. During the remote faculty observation, two observers simultaneously viewed the resident being precepted, assessing the faculty member using a standardized tool. After the experience, all participants were asked to complete a survey on the usability of the technology.

Results: Nineteen observations were successfully conducted from November 2011 to December 2012. From a qualitative perspective, faculty accepted this as a viable means of faculty development. Minor technical hurdles were captured in the survey and improved upon as staff and faculty became more comfortable with the technology and as our technical capabilities allowed. Overall, the technology was rapidly accepted into the practices.

Conclusions: Video teleconferencing represents a valuable tool that contributes to the development of faculty by making observation available to numerous sites, including remote areas that may have been previously challenging to reach due to logistics. Recent improvements in technology should make the process easier and allow more aspects of the encounters to be readily observed.

(Fam Med 2015;47(10):807-10.)

Video teleconferencing technology (telemedicine) has been used in trauma care,1 in assessing neonatal resuscitation skills,2 for telementoring during surgical and other procedures,3,4 and with standardized patients in the continuing education setting.5,6 Research has focused on the use of telemedicine technology to assess inter-professional skills;7,8 however, we found no substantive exploration of the role of video teleconferencing technology for faculty development.

Observation of learners during medical school clerkship and residency has been reviewed in the medical education literature.9-13 Studies have demonstrated that “the value of being clinically observed increases learner comfort and ability in the areas of physical examination and history taking.”14 However, without third-party observation and assessment of the interaction between faculty and students, faculty members do not receive the same quality of assessment regarding their teaching skills.

Training faculty in teaching skills, critical to the professional development of clinician-educators, has been shown to be effective.15-16 A 2004 survey revealed that only 39% of teaching hospitals had ongoing faculty development programs specific to teaching skills with <50% of the faculty participating.17 Three barriers exist to faculty development: lack of institutional support, teachers’ attitudes and misconceptions, and minimal research on what constitutes effective teaching improvement.18 While most faculty development occurs either through self-assessment or direct observation, faculty skills are poorly represented through self-assessment.16 In 2002, our institution conducted research to evaluate faculty teaching skills through direct observation with written feedback to improve learner-centered teaching and incorporate microskills use to teach patient-centered care.19 Results showed that faculty who were observed and received written feedback ranked higher in teaching skills than their non-observed faculty counterparts.20 Additional work has shown that developing outcome-based evaluation improves teaching effectiveness.21 Given the potential of telemedicine technology, the authors sought to test the practicality of utilizing this technology across multiple locations while conducting faculty development of preceptors.

 

Methods

 
 

This protocol was submitted to our organization’s Institutional Review Board (IRB) and deemed program evaluation; thus, the project did not undergo IRB review.

The eligible population of faculty included all preceptors engaged in teaching residents in five outpatient continuity care clinics of one family medicine residency program (six residents per year in a 3-year program) during the period from November 2011 to December 2012 (maximum eligible faculty n=28). For initial feasibility testing (Phase 1), a minimum of one core faculty physician was selected from each site (n=5) with the stipulation of having participated in faculty development in the Department of Family Medicine during the previous 2 years.

Key personnel at each site included faculty selected for Phase 1 and practice managers. Both were taught to use the telemedicine technology, which consisted of a Tandberg Edge 95 videoconferencing unit (Cisco Systems Inc, San Jose, CA), high-definition camera, 19” high-definition monitor, an uninterruptible power supply, and, in most locations, a heavy-duty mobile AV cart. At least 1 week prior to data collection, faculty participants were briefed on the protocol for the study and the tools.

In the second phase, two trained investigators observed faculty precepting residents during the first resident encounter available, regardless of postgraduate year. Faculty selection was based on faculty and observers’ availability. The videoconferencing equipment was placed outside of the patient room for the precepting session (standard procedure). Observers completed observation forms modeled on previous research20 (Figure 1), held a consensus discussion, and provided verbal feedback to the faculty member after the session. Participants were assured that data collected would not be used for job evaluation. Each observer sent a written observation form to the faculty for their records. The audiovisual material was viewed in real-time and was not recorded. No patient or resident information was collected.

Figure1

Following the observation and feedback session, faculty and observers assessed technology efficacy by completing a Computer System Usability Questionnaire (CSUQ),22 which included 19 questions using a 5-point Likert scale ( “disagree,” “somewhat disagree,” “neutral,” “somewhat agree,” “strongly agree”) and two open-ended questions regarding the most negative and positive aspects of using the system. Residents were not surveyed, as they did not interface with the technology.

 

Results

 
 

Two simultaneous observers monitored 19 faculty preceptors for one observation per preceptor from November 2011 through December 2012 across five outpatient practice locations. Each observation lasted an average of 21 minutes (range, 8–60 minutes). Longer observations typically involved cases in which the resident first discussed the case; next, the patient was seen off-camera; finally, precepting was completed on-camera.

Occasional problems with technology included poor connectivity or difficulty getting started; the latter was rapidly resolved with coaching from the site manager.

Five observers and 11 preceptors (n=16) completed the online CSUQ questionnaire (Table 1). Eight people responded to the open-ended questions (optional)—some listing as many as five responses to each positive/negative aspect (Table 2). The most negative aspects to the units: connectivity issues, lack of ease of use, or adequate training. Positive feedback included sound quality and saving on travel time.

Table1
Table2
 

Discussion

 
 

Previous studies have shown the benefit of using direct observation to assess learners10,11,13 as well as teachers;9,15,20 however, we were unable to find any literature describing the use of telemedicine technology for faculty development. Although our study was limited by small sample size, we demonstrated that telemedicine technology is feasible for assessing faculty at remote locations. We anticipated technology problems between the user with the interface and the device. Inexperience with the equipment caused Phase I participants some difficulties, which were resolved by an on-site guide and a designated site leader. Unlike other studies, both verbal and written feedback were given. One distinct advantage of the technology is the observer’s capacity to do other work while waiting for the observable moment.

Technology has improved since the implementation of this study to the point where a well-positioned iPad (Apple, Inc, Cupertino, CA) or a high-definition webcam connected to a laptop would suffice for remote observation. Key concerns in using telemedicine technology for observation are privacy regarding HIPAA-protected data, reliability of wireless networks throughout facilities, and maintaining secure video files. We guaranteed privacy by using a secure network between facilities and telemedicine units.

After this study, an online learning module was created to train medical educators in adopting telemedicine for direct observation of teaching skills. Implications include application to other specialties and locations where faculty development was previously not feasible. Appropriate consent from patients could lead to direct observation of the patient encounter, providing insights about faculty modeling of behavior and resident feedback. We plan to continue telemedicine observations, with the frequency driven by residency program/clerkship leaders and other stakeholders.

Acknowledgments: Funding/Support Disclosure: HRSA Award D76HP20861-01-00 of HRSA-10-186 Physician Faculty Development Grant and the Dorothy Rider Pool Health Care Trust.

The authors thank Sue Mathieu for participation in observation and data collection, Jacqueline Grove for editorial assistance, and Colin Coakley for assistance in securing the HRSA grant.

Corresponding Author: Address correspondence to Ms Biery, Lehigh Valley Health Network, Department of Family Medicine, One Center City, 8th Floor, PO Box 1806, Allentown, PA 18105-1806. 484-862-3068. Fax: 484-862-3096. nyann.biery@lvhn.org.

 

References

 
 
  1. Smith RS. Telemedicine and trauma care. South Med J 2001;94(8):825-9.
  2. Cronin C, Cheang S, Hlynka D, Adair E, Roberts S. Videoconferencing can be used to assess neonatal resuscitation skills. Med Educ 2001;35(11):1013-23.
  3. Rosser JC Jr, Herman B, Giammaria LE. Telementoring. Semin Laparosc Surg 2003;10(4):209-17.
  4. Panait L, Rafiq A, Tomulescu V, et al. Telementoring versus on-site mentoring in virtual reality-based surgical training. Surg Endosc 2006;20(1):113-8.
  5. Backstein D, Reznick R. Standardised patients and teleconferencing for CME: combining innovative ideas. Med Educ 2003;37(12):1066-7.
  6. Clever SL, Novack DH, Cohen DG, Levinson W. Evaluating surgeons’ informed decision making skills: pilot test using a videoconferenced standardised patient. Med Educ 2003;37(12):1094-9.
  7. Weinstein RS, López AM, Barker GP, et al. Arizona Telemedicine Program Interprofessional Learning Center: facility design and curriculum development. J Interprof Care 2007;21(Suppl 2):51-63.
  8. Careau E, Vincent C, Noreau L. Assessing interprofessional teamwork in a videoconference-based telerehabilitation setting. J Telemed Telecare 2008;14(8):427-34.
  9. Katz NT, McCarty-Gillespie L, Magrane DM. Direct observation as a tool for needs assessment of resident teaching skills in the ambulatory setting. Am J Obstet Gynecol 2003;189(3):684-7.
  10. Fromme HB, Karani R, Downing SM. Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med 2009;76(4):365-71.
  11. Howley LD, Wilson WG. Direct observation of students during clerkship rotations: a multiyear descriptive study. Acad Med 2004;79(3):276-80.
  12. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65(9 Suppl):S63-7.
  13. Ross V, Mauksch L, Huntington J, Beard M. Interdisciplinary direct observation: impact on precepting, residents, and faculty. Fam Med 2012;44(5):318-24.
  14. Jouriles N, Burdick W, Hobgood C. Clinical assessment in emergency medicine. Acad Emerg Med 2002;9(11):1289-94.
  15. Holmboe ES. Faculty and the observation of trainees’ clinical skills: problems and opportunities. Acad Med 2004;79(1):16-22.
  16. Wilkerson L, Irby DM. Strategies for improving teaching practices: a comprehensive approach to faculty development. Acad Med 1998;73(4):387-96.
  17. Clark JM, Houston TK, Kolodner K, Branch WT Jr, Levine RB, Kern DE. Teaching the teachers: national survey of faculty development in departments of medicine of US teaching hospitals. J Gen Intern Med 2004;19(3):205-14.
  18. McLean M, Cilliers F, Van Wyk JM. Faculty development: yesterday, today and tomorrow. Med Teach 2008;30(6):555-84.
  19. Parrott S, Dobbie A, Chumley H, Tysinger JW. Evidence-based office teaching—the five-step microskills model of clinical teaching. Fam Med 2006;38(3):164-7.
  20. Regan-Smith M, Hirschmann K, Iobst W. Direct observation of faculty with feedback: an effective means of improving patient-centered and learner-centered teaching skills. Teach Learn Med 2007;19(3):278-86.
  21. Hewson MG, Copeland HL. Outcomes assessment of a faculty development program in medicine and pediatrics. Acad Med 1999;74(10 Suppl):S68-71.
  22. Lewis JR. IBM Computer Usability Satisfaction Questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact 1995;7(1):1:57-78.

From the Lehigh Valley Health Network, Allentown, PA (Ms Biery and Dr Smith); Jump Simulation and Education Center, Peoria, IL. (Lehigh Valley Health Network at the time of the study) (Dr Bond); University of South Florida Morsani College of Medicine, Tampa, FL (Dr Smith); B. Braun Medical Inc., Bethlehem, PA (Lehigh Valley Health Network at the time of the study) (Mr LeClair); College of Communication, DePaul University, Chicago, IL (Lehigh Valley Health Network at the time of the study) (Dr Foster).

 


Copyright 2018 by Society of Teachers of Family Medicine