Printed from: http://www.stfm.org/NewsJournals/FamilyMedicine/PastIssues/ArticleView
FMHubTOC

Are Self-study Procedural Teaching Methods Effective? A Pilot Study of a Family Medicine Residency Program

Brandy Deffenbacher, MD; Shannon Langner, MD; Morteza Khodaee, MD, MPH

Background and Objectives: A family medicine residency is a unique training environment where residents are exposed to care in multiple settings, across all ages. Procedures are an integral part of family medicine practice. Family medicine residency (FMR) programs are tasked with the job of teaching these skills at a level of intensity and frequency that allows a resident to achieve competency of such skills. In an environment that is limited by work hour restrictions, self-study teaching methods are one way to ensure all residents receive the fundamental knowledge of how to perform procedures. We developed and evaluated the efficacy of a self-study procedure teaching method and procedure evaluation checklist.

Methods: A self-study procedure teaching intervention was created, consisting of instructional articles and videos on three procedures. To assess the efficacy of the intervention, and the competency of the residents, pre- and postintervention procedure performance sessions were completed. These sessions were reviewed and scored using a standardized procedure performance checklist.

Results: All 24 residents participated in the study. Overall, the resident procedure knowledge increased on two of the three procedures studied, and ability to perform procedure according to expert-validated checklist improved significantly on all procedures.

Conclusions: A self-study intervention is a simple but effective way to increase and improve procedure training in a way that fits the complex scheduling needs of a residency training program. In addition, this study demonstrates that the procedure performance checklists are a simple and reliable way to increase assessment of resident procedure performance skills in a residency setting.

(Fam Med. 2017;49(10):789-95.)

Graduates of family medicine residencies are expected to be competent in a variety of procedures. In 2008, the Society of Teachers of Family Medicine (STFM) Group on Hospital and Procedural Training developed a list of procedures within the scope of family medicine, and in 2009 created an advanced procedure list.1, 2 The current Accreditation Council for Graduate Medical Education (ACGME) Family Medicine Requirements state that “residents must receive training in clinic procedures required for their future practices in ambulatory and hospital environments” and that each program is required to develop this list, as well as document objective procedure competence of each resident.3 Procedural skills are not being adequately taught.4-6 To learn a procedure, one must see a demonstration, practice, and have time to develop autonomy.7,8 The busy resident schedule and current work hour restrictions create challenges in ensuring that all residents have the same opportunities to learn, observe, and perform procedures.9

While many methods are used to assess resident clinical skills, including procedural skills, few have been evaluated for validity.10 Programs have relied on logbooks to track resident numbers, but these are not descriptive, nor do they track progress towards attaining competence.8 Direct observation using a global and task-specific tool is a comprehensive approach that should be used when assessing competence in procedural skills,11 however, few of these assessment methods and tools have been validated.10, 12

Given complex schedules, and the difficulty in providing procedure skills training to all residents, a self-study intervention was developed. This paper presents an educational intervention and the evaluation method used to assess resident performance.

 

Methods

 
 

This study used a pretest/posttest experimental research design on family medicine residents at the University of Colorado during the 2009-2010 academic year that was approved by the Colorado Multiple Institutional Review Board. Three procedures (intrauterine device [IUD] insertion, knee aspiration, and skin punch biopsy) were selected based on the frequency with which they were performed, faculty comfort level in teaching them, and resident interest in performing these procedures after graduation (Figure 1).13 A procedure kit was assembled for each procedure that included a model of the body area (task trainer), and an equipment tray that included the tools needed to perform the procedure. A quiz consisting of multiple choice and true/false questions was written to assess residents’ knowledge of the procedures (Table 1). Procedure performance evaluation checklists were created that included all the steps to prepare the patient for the procedure, perform the procedure, and provide postprocedure counseling (Table 2). Previous procedural competency instruments were used as templates.14 We used content validity for validating these instruments using a group of experts in our faculty team. The procedure kits were set up in a procedure room with a camera to record the session. A self-study intervention was developed that consisted of instructional articles15-17 reviewing each procedure and a link to videos demonstrating how to perform each one.16-18 Each resident signed up for recording sessions.

 

figure1

 

 

Table 1

 

table2

 

During the preintervention sessions, the resident was instructed to complete the knowledge quiz, and then perform each procedure as if they were performing it on an actual patient. After completing the presession, the residents were emailed the self-study education intervention.18-20 Four weeks after the preintervention session, the residents completed the postintervention session, which was exactly the same as the preintervention session. Upon completion of the pre- and postintervention sessions, the knowledge quizzes were scored and the authors reviewed the videos separately and completed the procedure performance evaluation checklists using a scale of 1: not competent, 2: partially competent, and 3: fully competent, as shown in Table 2.

We conducted statistical analyses with the IBM SPSS, version 23.0 (SPSS, Inc, Chicago, Illinois). We used descriptive statistics to examine the frequency, mean, median, and range for all variables. A Wilcoxon matched-pairs test for paired samples was used to assess the impact of the intervention on the data obtained in the two predetermined moments (pre- and postintervention).

 

Results

 
 

All 24 family medicine residents (eight PGY-1s, nine PGY-2s, and 7 PGY-3s) participated. Average procedure knowledge (Figure 2) on two of the three procedures (IUD insertion and punch biopsy) improved (Wilcoxon P=0.617, P=0.006, and P=0.016, respectively). After compiling all the procedure performance checklist scores, residents’ procedure competency improved (Figure 3). The average competency score in knee arthrocentesis, IUD insertion, and punch biopsy significantly improved (Wilcoxon P=0.000, P=0.000, and P=0.000, respectively).

 

figure2

 

figure3

 

 

Discussion

 
 

There are many barriers that educators encounter when trying to teach procedures, including work hour restrictions, lack of a structured procedure curriculum, and absence of validated competency assessment tools. Workshops are a common teaching method, but are effective only when participants are able to practice those skills shortly after they are learned.5 While workshops are still a valid teaching modality, self-study interventions are needed to accommodate work hour restrictions that can prevent residents from attending them.9 A recent study compared the traditional in-person workshop to video module instruction for teaching IUD insertion, and it demonstrated that a video module was an effective and reliable method.21 Our study demonstrates that a self-study intervention can improve resident knowledge and ability to perform a procedure.

Residents consistently express the desire for structured feedback. Programs struggle with providing this direct observation and formative feedback.8 The competency checklists could help meet the needs of both residents and programs alike. These checklists allow the program to assess the skill level of each resident, provide a standardized procedure teaching method across the family medicine residency, and allow the resident to get structured feedback when used in a simulation or real patient encounter. As residency programs move from paper tracking systems to online programs and applications to track resident performance, validated procedure competency checklists should be included.

Despite all of the residents in the program participating in the study, one of the limitations of this study is its sample size. Our study serves as a proof-of-concept demonstrating that this modality of teaching and assessment is feasible. Additional studies would be required to look at knowledge retention and skill decay rates following educational interventions. Such studies would allow programs to look at frequency of educational interventions and deliberate practice sessions to attain and maintain competency. While this study demonstrates the effectiveness of the self-teaching intervention 1 month after completion of the preintervention session, it does not take into consideration when the resident completed the intervention, nor any exposure to, nor performance of, the procedures between the pre- and postintervention sessions. Recall bias is a limitation of the study design. It is possible that the improvement in the knowledge quiz scores has more to do with exposure to the previous quiz than the self-study intervention. Further large-scale studies with interrater reliability evaluation can help standardize these educational tools.

Our study demonstrates that a self-study intervention is an effective method for teaching procedures that can be used in combination with other teaching modalities. The performance checklists can help standardized procedure training, increase the amount of formative feedback residents receive, as well as track and document residents’ procedure performance and eventual attainment of competence.

Corresponding Author: Address correspondence to Dr Deffenbacher, Assistant Professor, University of Colorado School of Medicine, Department of Family Medicine, 3055 Roslyn Street Suite100, Denver, CO 80238. 720-553-2666.
Brandy.Deffenbacher@ucdenver.edu.

 

References

 
 
  1. Kelly BF, Sicilia JM, Forman S, et al. Advanced procedural training in family medicine: a group consensus statement. Fam Med. 2009; 41(6):398-404.
  2. Nothnagle M, Sicilia JM, Forman S, et al. Required procedural training in family medicine residency: a consensus statement. Fam Med. 2008; 40(4):248-252.
  3. ACGME Program Requirements for Graduate Medical Education in Family Medicine. 2016. Available at: http://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/120_family_medicine_2016.pdf Accessed January 25, 2017.
  4. Gaies MG, Landrigan CP, Hafler JP, et al. Assessing procedural skills training in pediatric residency programs. Pediatrics. 2007; 120(4):715-722.
  5. MacKenzie MS, Berkowitz J. Do procedural skills workshops during family practice residency work? Can Fam Physician. 2010; 56(8):e296-301.
  6. Wickstrom GC, Kelley DK, Keyserling TC, et al. Confidence of academic general internists and family physicians to teach ambulatory procedures. J Gen Intern Med. 2000; 15(6):353-360.
  7. Norris TE, Cullison SW, Fihn SD. Teaching procedural skills. J Gen Intern Med. 1997; 12 Suppl 2:S64-70.
  8. Touchie C, Humphrey-Murto S, Varpio L. Teaching and assessing procedural skills: a qualitative study. BMC Med Educ. 2013; 13:69.
  9. Choby B, Passmore C. Faculty perceptions of the ACGME resident duty hour regulations in family medicine. Fam Med. 2007; 39(6):392-398.
  10. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. J Am Med Assoc. 2009; 302(12):1316-1326.
  11. Ahmed K, Miskovic D, Darzi A, et al. Observational tools for assessment of procedural skills: a systematic review. Am J Surg. 2011; 202(4):469-480 e466.
  12. Morris MC, Gallagher TK, Ridgway PF. Tools used to assess medical students competence in procedural skills at the end of a primary medical degree: a systematic review. Med Educ Online. 2012; 17.
  13. Langner S, Deffenbacher B, Nagle J, et al. Barriers and methods to improve office-based procedural training in a family medicine residency. Int J Med Educ. 2016; 7:158-159.
  14. Goldstein DJ et al. Procedure Competency Assessment Tools (PCATs). http://resourcelibrary.stfm.org/viewdocument/procedure-competency-assessment-too. Accessed January 25, 2017.
  15. Johnson BA. Insertion and removal of intrauterine devices. Am Fam Physician. 2005; 71(1):95-102.
  16. Zuber TJ. Knee joint aspiration and injection. Am Fam Physician. 2002; 66(8):1497-1500, 1503-1494, 1507.
  17. Zuber TJ. Punch biopsy of the skin. Am Fam Physician. 2002; 65(6):1155-1158.
  18. Medical Videos Service. Mole Removal Using Punch Biopsy. https://www.medicalvideos.us/video/1592/mole-removal-using-punch-biopsy.Accessed August 31, 2017.
  19. Medical Videos Service. Intra-Uterine Device IUD Insertion. https://www.medicalvideos.us/video/3316/intra-uterine-device-iud-insertion. Accessed August 31, 2017.
  20. Thomsen TW, Shen S, Shaffer RW, et al. Videos in clinical medicine. Arthrocentesis of the knee. N Engl J Med. 2006; 354(19):e19.
  21. Garcia-Rodriguez JA, Donnon T. Using comprehensive video-module instruction as an alternative approach for teaching IUD insertion. Fam Med. 2016; 48(1):15-20.

From the University of Colorado School of Medicine, Department of Family Medicine.


Copyright 2018 by Society of Teachers of Family Medicine