Printed from:

Integrative Medicine in Residency: Feasibility and Effectiveness of an Online Program

Patricia Lebensohn, MD; Benjamin Kligler, MD, MPH; Audrey J. Brooks, PhD; Raymond Teets, MD; Michele Birch, MD; Paula Cook; Victoria Maizes, MD

Background and Objectives: Online curricular interventions in residency have been proposed to address challenges of time, cost, and curriculum consistency. This study is designed to determine the feasibility and effectiveness of a longitudinal, multisite online curriculum in integrative medicine (IMR) for residents.

Methods: Residents from eight family medicine programs undertook the 200-hour online IMR curriculum. Their medical knowledge (MK) scores at completion were compared to a control group from four similar residency programs. Study and control groups were comparable in baseline demographics, and MK scores. Course completion, MK scores, and course evaluations were assessed.

Results: Of 186 IMR residents, 76.9% met completion requirements. The IMR group showed statistically significant higher MK scores at residency completion, the control group did not (IMR: 79.2% vs. Control: 53.2% mean correct). Over three-fourths of IMR participants (range 79-92%) chose the top two rating categories for each course evaluation item. In an exit survey, ability to access the curriculum for 1 additional year and intention to utilize IM approaches after residency were the highest ranked items.

Conclusions: The demonstrated feasibility, effectiveness, and positive evaluations of the IMR curriculum indicate that a multisite, online curricular intervention is a potentially viable approach to offering new curriculum with limited on-site faculty expertise for other family medicine residencies.

(Fam Med. 2017;49(7):514-21.)

In 2007, in collaboration with faculty from eight family medicine residencies around the country the University of Arizona Center for Integrative Medicine launched the Integrative Medicine in Residency (IMR) program. The Arizona Center for Integrative Medicine defines integrative medicine as healing-oriented medicine that takes account of the whole person, including all aspects of lifestyle. It emphasizes the therapeutic relationship between practitioner and patient, is informed by evidence, and makes use of all appropriate therapies. The purpose of the IMR was to develop a high-quality, web-based curriculum in integrative medicine (IM) (including, but not limited to, complementary and alternative medicine (CAM)) to facilitate the incorporation of training in IM as a routine part of family medicine residency education. A description of the IMR curriculum, development process and implementation have been previously published.1 This article presents the results of our 5-year pilot phase, the impact of the program on residents’ IM knowledge, and the experience of creating and evaluating an online multisite collaborative curriculum.

According to the Centers for Disease Control, 40-45% of adults in the US use CAM.2 The Institute of Medicine (IOM) recommends that “health profession schools (eg, schools of medicine, nursing, pharmacy, and allied health) incorporate sufficient information about CAM into the standard curriculum at the undergraduate, graduate, and postgraduate levels to enable licensed professionals to competently advise their patients about CAM.”3 The Society of Teachers of Family Medicine (STFM) endorsed the inclusion of CAM material in family medicine residency training in 1999. Published curriculum guidelines are available for medical school4 and residency education.5 A recent national survey found that family medicine programs offering IM curriculum had significantly higher pass rates on the American Board of Family Medicine (ABFM) board exam.6 However, possibly because of competing curricular priorities and a lack of residency faculty qualified to teach on this subject, these guidelines have not been widely implemented, and the majority of residency programs do not offer a formal IM curriculum.7 Consequently, many family medicine residents feel underprepared to adequately counsel patients in this area.8 The IMR was developed to address challenges of effective implementation of these guidelines with input from family medicine program directors, faculty, and residents in order to provide standardized and high-quality training in this area.9

The IMR, now adopted at over 60 residencies in the United States and Canada, is a 200-hour curriculum delivered in stages over the 3 years of residency. Roughly 85% of the curriculum (186 hours) is web-based, with the remainder consisting of unique hands-on, on-site activities based on local resources, eg, cooking classes, botanical preparations, movement therapies, experiential mind-body activities, etc. Modular courses can be placed flexibly to meet institutional needs, with the content designed to complement existing didactic curriculum. The online curriculum content is highly interactive and covers areas not well addressed in medical education including nutrition, physical activity, mind-body medicine, spirituality, environmental health, and CAM interventions (eg, botanicals, manual medicine, Chinese medicine). Courses are case based with emphasis on preventive medicine and care of chronic medical conditions. Curriculum modules and completion time estimates are presented in Table 1.



Mentored online distributed learning addresses many of the challenges residency programs experience with scheduled lectures and duty hours. The flexibility provided by online curriculum allows residents to access material at their convenience (nights, weekends, slower rotations), addressing the challenge of busy schedules when lectures are often missed to attend to urgent clinical demands. Importantly, a distributed learning model may ultimately prove to be more cost-effective and offer greater curricular consistency than traditional didactic-based educational programming.

Although many have speculated that the “impersonal” nature of web-based learning (WBL) would result in poorer performance, studies show the contrary, perhaps because of the increased ability to standardize content and teaching strategies as well as active engagement in a WBL approach.10-11 Multiple studies have shown online learning in residency education is equivalent or superior to traditional didactic-based approaches in terms of residents’ knowledge and competency outcomes. Criley et al used online training to supplement education around the cardiac exam for family and internal medicine residents (n=59). In comparison with traditional methods, they found post-test improvement immediately after curriculum completion, as well as at follow-up, suggesting the web-based intervention led to content retention.12 In a cluster randomized trial of internal and family medicine residents (n=129) evaluating an online intervention to improve attitudes and communication skills with patients with substance use disorders, Lanken et al found significantly improved outcomes in communication skills on standardized patient encounters in the online education group.13 Sperl-Hillen, et al found that an online case-based virtual patient simulation to improve diabetes management tested in primary care residents (n=341) resulted in significantly improved knowledge and skills compared to standard training.14 Evaluations of online curricula in radiation oncology15 and geriatric16 residencies have shown similar results. In addition to consistently leading to better outcomes in terms of skills, knowledge, and competency evaluations, some studies have found online education in residency is preferred over traditional formats.17

These examples suggest that online learning offers some advantages over traditional didactics, while providing for equivalent medical knowledge improvements. However, to date, no program of the scope and duration of the IMR program has been widely implemented and rigorously evaluated. The IMR curriculum intervention was 3 years in duration, offered at eight residencies for three consecutive classes. It included a control group for comparison purposes, and provided an evaluation structure built into the online curriculum. This paper describes the feasibility and effectiveness of this effort to bring online education in residency to this new level, both in terms of the duration and scope of the curriculum, and evaluation rigor.





Two hundred twelve residents started family medicine training at eight residencies offering the IMR in the 2008-2010 incoming classes. Participating residencies were: University of Arizona; Beth Israel Medical Center, New York, NY; Carolinas Medical Center, Charlotte, NC; Hennepin County Medical Center, Minneapolis, MN; Maine-Dartmouth, Augusta, ME; Maine Medical Center, Portland, ME; University of Connecticut, Hartford, CT; and University of Texas Medical Branch, Galveston, TX. The pilot sites were chosen to represent the heterogeneity seen in family medicine residency programs, representing community and academic settings, and urban, suburban and rural locations. Twenty-six of the 212 residents (12.3%) did not complete the residency program and,therefore, did not participate in the IMR program. The graduating sample consisted of 186 residents across the 3 classes in the 8 programs. The evaluation results presented are for the graduating sample.

Residents in the 2009 and 2010 classes from four family medicine residency programs without the IMR were recruited to serve as a control group (N=53). Control programs had characteristics similar to IMR sites, competing with IMR sites for residency applicants. Participating control residencies were: Montefiore Medical Center, Bronx, NY; Moses Cone Memorial Hospital, Greensboro, NC; Providence Hospital, Anchorage, AK; and University of Minnesota, Minneapolis, MN. Forty-six (86.8%) completed at least one assessment and 32 (60.4%) completed assessments at graduation. Residents were compensated for study participation.

Residents in the IMR programs were primarily female (61.3%), Caucasian (58.5%), married or cohabitating (58.5%), and an average age of 30.4 years (range 24-50 years old). Residents in the control group programs were comparable demographically: 60.4% female, 65.4% Caucasian, 66% married or cohabitating, average age 29.9 years (range 25-53 years old).


Course completion. Residents were required to complete at least 80% of the 186-hour online curriculum.

Medical knowledge. The final medical knowledge (MK) test had 58 multiple choice questions based on course objectives and covering the entire curriculum content. Ongoing curriculum updates to address residents’ interest and priorities necessitated changes in the exam during the first 3 years. Ongoing psychometric analyses using Classical Test Theory were conducted prior to 2011 graduation to ensure a valid final test for the first class. Poorly performing items were identified and revised or dropped from subsequent test versions. Psychometric analyses were repeated for the 2011 and 2012 graduating classes to monitor test validity. IMR residents completed the MK test in PG years 1, 2, 3 and upon residency completion; control group residents completed the MK test in PG years 1, 2 and upon residency completion. MK test results from PG year 1 and residency completion are presented.

IMR completion. IMR completion was defined as completing at least 80% of the online courses and passing the final MK test with a score of 70% or higher. Residents were given three attempts to pass the MK test.

Course Evaluation

An evaluation survey was administered at the end of each course to assess achievement of course objectives, clinical utility of the content, educational complexity/depth, usefulness of course resources, and smoothness of the online technology. Items were rated on a 5-point scale and averaged across all courses. The percentage of course participants endorsing the top two categories are presented. A 7-item exit survey rating curriculum experience, faculty/program support, and relevance to career was administered upon completion of the curriculum. Items were rated on a 5-point scale (strongly disagree to strongly agree). The mean rating is presented.

Site characteristics survey. A site survey was created to assess how the IM online curriculum was implemented across the pilot sites. The survey examined IM culture, faculty characteristics, additional IM on-site activities (Table 2) and whether IMR completion was required for graduation. Pilot sites were surveyed when the third class of residents graduated (2013). The number of characteristics endorsed within each survey area was summed for each site.



Statistical Approach

Descriptive statistics are presented for course completion, MK score, IMR completion, course evaluation, and site characteristics. Separate Chi-square analyses were conducted to examine IMR completion by each area in the site characteristics survey. Repeated measures general linear models were conducted comparing change in MK between the start and completion of residency by group. Analyses were conducted using IBM® SPSS® Statistics Desktop V21.0 (Armonk, New York).

Approval for the study was granted by the University of Arizona Institutional Review Board (IRB) and IRBs of the pilot and control sites.




Course Completion

Average completion of the online curriculum content was 90.5%, ranging from 0%-100% completion (n=186). Of the 186 residents, 166 (88.2%) completed at least 80% of the online curriculum, meeting the course completion requirement for IMR completion.

Medical Knowledge

The average MK score at graduation was 79.1%, range 42%-95% (n=151). Thirty-five of the 186 residents did not take the final MK test (18.8%). Of the 151 residents who took the final MK test, 142 (94%) received a score of 70% or higher, meeting the medical knowledge completion requirement for IMR completion. Post-hoc comparisons utilizing Bonferroni correction showed a statistically significant difference between residency start and graduation MK scores for the IMR group (P<0.001), but not the control group. A statistically significant difference between groups at graduation was also found (P<0.001, Figure 1). Due to missing data at residency start for 17 of the 151 IMR residents and variation between classes on the PG year 1 test content, a t-test comparing IMR and control group residents at graduation was conducted. Residents participating in the IMR scored significantly higher on the MK test than control group residents at the end of residency (t(181)=15.1; P<0.001; Mean±SD 79.1%±9.0 vs 53.2%±8.0). Since IMR residents were given three attempts to pass the test, while control group residents received one attempt, IMR first attempt MK scores were compared to the control group MK scores also. When comparing first attempts, IMR residents scored significantly higher than control group residents (t(181)=10.3; P<0.001; Mean±SD 74.6%±11.2 vs 53.2%±8.0). Most IMR residents passed the test on the first attempt (n=113/151; 75%). Of the 38 residents who did not pass on their first attempt, 24 passed on their second attempt, five passed on their third attempt, and nine residents did not retake the test after not passing on their first or second attempt.




IMR Completion

Overall, 76.9% (n=143) of the 186 graduating residents received a certificate of completion; 139 met both IMR completion requirements, passing the medical knowledge test and completing at least 80% of the online curriculum, and the other four were granted the certificate under special conditions. Faculty leaders advocated that these four residents were eligible to receive the IMR certificate. Three residents passed the MK test but completed less than 80% of the curriculum content (74%, 78%, and 79%). One completed 100% of the curriculum content but did not pass the MK test (67%). Of the 43 residents who did not meet the IMR completion, 26 completed 80% of the curriculum, however, they either did not take (n=18) or pass the MK test (n=8). The remaining 17 residents did not complete 80% of the curriculum or take the MK test.

Course Evaluation

The percent of participants choosing the top two rating categories for course evaluation items ranged from 79%-92%, with “meeting course objectives” receiving the highest rating and “educational depth” receiving the lowest rating (Figure 2). Average ratings for the IMR exit survey ranged from 3.7-4.4. The highest-rated item was an additional year’s access to the IMR curriculum. This was followed closely by intention to utilize IM approaches in future practice and continue IM education. The two lowest-rated items concerned faculty support for completing the IMR material and relating IM curriculum materials to residency training (Figure 3).



Site Characteristics

The total number of IM culture characteristics ranged from two to four at a site, with all sites offering an IM consultation (Table 2). Having an IM fellowship (37.5%) and dually accredited MD/DO residency (25%) were the least frequent characteristics across the sites. Five sites (62.5%) had both faculty leader fellowship-trained and faculty leader with designated IM teaching time. Additional IM activities ranged from two to four per site, with all sites offering IM teaching rotation/IM electives and support for residents applying IM knowledge in clinics. Three sites required IMR completion for graduation each year, one site required it for two classes and another site required it for the last graduating class.

The number of additional activities offered to support IM (χ2(2)=32.04; P<0.001) and whether IMR completion was required for graduation (χ2(2)=33.87; P<0.001) were associated with IMR completion. IMR completion rate was higher at sites with either three or four additional IM activities (92%; 85% completed respectively) vs sites with two additional activities (51% completed). There was a 95% resident completion rate when IMR completion was required for graduation vs a 59% completion rate when it was not required. IM culture characteristics and faculty characteristics were not associated with IMR completion.




Despite the IOM’s charge to increase knowledge among the health professions in complementary and alternative medicine, and STFM’s guidelines regarding this, to date there has not been effective widespread implementation in residency education. The IMR goal was to offer a core IM curriculum that was easily integrated into residency training. Designing an online curriculum for use across multiple sites with varying requirements and resources dictates flexibility in accepting variation in implementation and results. Despite the variation, the evaluation of the 5-year pilot phase of the IMR program showed that it was successfully integrated into the curriculum at all eight family medicine residency sites. It resulted in improved knowledge in IM compared to a control group and it was accepted and viewed favorably by a majority of residents.

Seventy-seven percent of residents met the IMR criteria for completion, despite not all programs requiring completion for graduation. Not surprisingly, programs requiring it had a significantly higher (95%) completion rate. In programs not requiring IMR completion, a significant number of residents completed ≥80% of the material but simply neglected to take the final test. Thus, the overall course completion rate across the entire pilot was 90.5%, an impressive figure which strongly supports the concept of residents as adult learners capable of carrying the responsibility of self-directed learning, even through the demands of residency.

The MK test provided objective evidence of residents’ learning in the IMR curriculum, both in comparison to their baseline scores and in comparison to the control group. These 58 items were chosen based on the major learning objectives of each course within the curriculum. As IM incorporates both conventional and CAM approaches to health and treatment of disease states, some items test general primary care knowledge and others IM-specific knowledge. It is not surprising that at the start of the IMR program both IMR and control groups were able to answer approximately half of the questions correctly. However, the control group failed to improve significantly while the IMR group’s scores rose to approximately 80% average correct at graduation.

In addition to making the IMR a graduation requirement, the other site characteristic significantly associated with a higher completion rate was the number of additional onsite IM activities offered to complement and reinforce online learning. Although all sites had some on-site activities associated with the IMR, having at least three activities led to a substantially higher completion rate. It is possible that the community-building and social support associated with in-person vs online activities, as well as a more effective integration of the IMR into the residency culture as a result of on-site activities, contributed to this effect.

“Blended learning” (combining traditional face-to-face education with synchronous or asynchronous online learning) has been shown to be at least as effective, if not more effective, than non-blended learning.18 Further research is needed to elucidate just how much “blending” is optimal to achieve effectiveness and maintain learner satisfaction.

The residents’ overall evaluations of the IMR curriculum were extremely positive, consistent with prior studies demonstrating the acceptability of online learning in medical education. Interestingly, residents highly valued maintaining free access to the IMR content for the year after graduation, when it was no longer required for residency. Residents also saw the curriculum as having high utility for their future practice and expressed desire to learn more IM. Residents gave low ratings for the degree to which faculty assisted them with completing IMR courses on time, suggesting that more faculty involvement in the online curriculum process as well as in organizing on-site activities would be optimal.


The control group for the IMR evaluation was comprised of volunteers who were paid to complete online self-administered assessments. Of the 53 original control group volunteers, approximately 87% completed one assessment and 60% completed the assessments at graduation. While there was no difference in baseline MK scores, it is unknown whether those completing both assessments differed from the overall original sample in terms of their experience with IM. Nor was it possible to control for any outside exposure or IM learning in the control group. Nonetheless, the equivalent medical knowledge scores at time zero and gains by only the IMR group point to an effect of the IMR curriculum.

A final important limitation in our evaluation is that although some programs implemented direct observation evaluation tools, such as standardized patient encounters and OSCEs, not all pilot sites were able to. In the future we hope to develop more rigorous skill and competency-based assessment strategies for use in IMR programs. Another proposed area of research is to examine post-graduation self-assessments of residents’ abilities and knowledge of IM techniques and applications to the medical conditions they encounter in their practices after residency.

Finally, a limitation of widely implementing the IMR beyond the initial pilot phase is its cost. In order to keep the content updated and have enough support staff to manage registration and technical support, as well as to provide faculty development to faculty leaders there is a yearly tuition that depends on the number of residents enrolled. We adapted the fee structure to accommodate programs with different numbers of users and discounted for longevity in the IMR program (from $5,000 for up to four users to $25,000 for unlimited users). Residencies use a number of creative strategies for funding the IMR, including having residents use CME money, including IMR in their operating budget, and seeking private foundations grants.




The demonstrated success of the IMR—which started at eight pilot sites, targeting only family medicine, and now being used in more than 60 programs including internal medicine, pediatrics and preventive medicine—suggests that our model of a collaborative multisite, web-based curriculum intervention is a potentially viable approach for primary care residencies in addressing a variety of new curricular domains. Our results show that requiring completion for graduation and providing a modest number of on-site activities to support the online curriculum may be important elements of successful implementation at a residency program.

Acknowledgements: The authors wish to thank Emily Sherbrooke for her invaluable assistance in preparing this manuscript.

Financial support: Funding for the Integrative Medicine in Residency program and its evaluation has been provided by a Congressionally-directed grant administered through the US Department of Education. Contents do not necessarily represent the policy of the Department of Education and do not assume endorsement by the Federal Government. Additional support comes from the Weil Foundation, the David C. and Lura M. Lovell Foundation, the Joan B. Diamond Charitable Lead Trust, and multiple generous individuals. None of the funders had any role in the design or conduct of the study, the collection, management, analysis, and interpretation of data, the preparation, review or approval of the manuscript, or the decision to submit the manuscript for publication.

Presentations: Partially reported at the 2015 STFM Annual Conference, Orlando, FL, and 2014 North American Primary Care Research Group Annual Meeting, New York, NY.

Corresponding Author: Address correspondence to Dr Lebensohn, Arizona Center for Integrative Medicine, University of Arizona, PO Box 245153, Tucson, AZ 85724. 520-626-9390. Fax: 520-626-6484.



  1. Lebensohn P, Kligler B, Dodds S, Schneider C, Sroka S, Benn R, Cook P, Guerrera M, Low Dog T, Sierpina V, Teets R, Waxman D, Woytowicz J, Weil A, Maizes V. Integrative medicine in residency education: developing competency through online curriculum training. J Grad Med Educ. 2012;3(1):76-82.
  2. Barnes P, Powell-Grier E, McFann K, Nahin R. Complementary and alternative medicine use among adults: United States, 2002. CDC Advance Data Report #343. May 27, 2004.
  3. Complementary and alternative medicine in the United States: Institute of Medicine Report [January 12, 2005]. ( Accessed 5/27/14.
  4. Kligler B, Maizes V, Schachter S, et al. Core competencies in integrative medicine for medical school curricula: a proposal. Acad Med. 2004;79(6):521–31.
  5. Kligler B, Gordon A, Stuart M, Sierpina V. Suggested curriculum guidelines on complementary and alternative medicine: recommendations of the Society of Teachers of Family Medicine Group on Alternative Medicine. Fam Med. 2000;32(1):30-3.
  6. Mims L, Mainous A, Chirina S, Carek P. Are specific residency program characteristics associated with the pass rate of graduates on the ABFM certification examination? Fam Med. 2014;46(5):360-8.
  7. Gardiner P, Filippelli AC, Lebensohn P, Bonakdar R. Family medicine residency program directors attitudes and knowledge of family medicine CAM competencies. EXPLORE: The Journal of Science and Healing 2013;9(5):29.
  8. Prajapati SH, Kahn RF, Stecker T, Pulley L. Curriculum planning: a needs assessment for complementary and alternative medicine education in residency. Fam Med. 2007;39(3):190-4.
  9. Benn R, Maizes V, Guerrera M, Sierpina V, Cook P, Lebensohn P. Integrative medicine in residency: assessing curricular needs in eight programs. Fam Med. 2009;41(10):708-14.
  10. Wiecha JM, Chetty VK, Pollard T, Shaw PF. Web-based versus face-to-face learning of diabetes management: the results of a comparative trial of educational methods. Fam Med. 2006;38(9):647-52.
  11. Schilling K, Wiecha J, Polineni D, Khalil S. An interactive web-based curriculum on evidence-based medicine: design and effectiveness. Fam Med. 2006;38(2):126-32.
  12. Criley JM, Keiner J, Boker JR, Criley SR, Warde CM. Innovative web-based multimedia curriculum improves cardiac examination competency of residents. J Hosp Med. 2008;3(2):124-33.
  13. Lanken PN, Novack DH, Daetwyler C, et al. Efficacy of an internet-based learning module and small-group debriefing on trainees’ attitudes and communication skills toward patients with substance use disorders: results of a cluster randomized trial. Acad Med. 2015;90(3):345-54.
  14. Sperl-Hillen J, O’Connor PJ, Ekstrom HL, et al. Educating resident physicians using virtual case-based simulation improves diabetes management: a randomized controlled trial. Acad Med. 2014;89(12):1664-73.
  15. Alfieri J, Portelance L, Souhami L, et al. Development and impact evaluation of an e-learning radiation oncology module. Int J Radiat Oncol, Biol, Phys. 2012;82(3):e573-80.
  16. Westmoreland GR, Counsell SR, Tu W, Wu J, Litzelman DK. Web-based training in geriatrics for medical residents: a randomized controlled trial using standardized patients to assess outcomes. J Am Geriatr Soc. 2010;58(6):1163-9.
  17. Cook DA, Dupras DM, Thompson WG, Pankratz VS. Web-based learning in residents’ continuity clinics: a randomized, controlled trial. Acad Med. 2005;80(1):90-7.
  18. Liu Q, Peng W, Zhang F, Hu R, Li Y, Yan W. The effectiveness of blended learning in health professions: systematic review and meta-analysis. J Med Internet Res. 2016;18(1):e2.

From the University of Arizona Center for Integrative Medicine, University of Arizona College of Medicine (Drs Lebensohn, Brooks, Maizes, and Ms Cook); Icahn School of Medicine at Mount Sinai (Dr Kligler); Albert Einstein School of Medicine, New York, NY (Dr Teets); Department of Family Medicine, Carolinas Medical Center, Charlotte, NC (Dr Birch).

Copyright 2017 by Society of Teachers of Family Medicine