Wendling AL, Wudyka AE, Phillips JP, Levine DL, Mulhem R, Neale AV, Morley CP. RU4PC? Texting to Quantify Feedback About Primary Care and Its Relationship With Student Career Interest. Fam Med 2015;48(1):21-29.
RU4PC? Texting to Quantify Feedback About Primary Care and Its Relationship With Student Career Interest
Andrea L. Wendling, MD; Andrea E. Wudyka, MD; Julie P. Phillips, MD, MPH; Diane L. Levine, MD;
Elie Mulhem, MD; Anne Victoria Neale, PhD, MPH; Christopher P. Morley, PhD, MA
Background: Faculty and residents routinely offer feedback about medical students’ specialty career preferences, yet influence of feedback has not been quantitatively evaluated. This study aims to report incidence and balance of primary care comments heard by medical students and determine the effect of comments on career interest.
Methods: This multicenter observational cohort study used text messaging and online surveys to examine positive and negative feedback about primary care, as reported by medical students in real time between September 2012 and April 2013. Participants from three universities sent short text messages when primary care comments were heard during two 30-day periods; each period was preceded and followed by surveys assessing career interest.
Results: A total of 120 students (86.3% of recruits) participated in at least one texting period; 87 (62.6%) participated in all aspects of study. Overall, positive comments (851) outnumbered negative (616). Total number of negative comments reported per student was associated with a significantly lower interest in primary care career (β=-.04). There was an association between students’ negative-to-positive comment ratios and lower interest in primary care that approached significance (β=-.145) but only became significant (β=-.191) when variables including institution were added to a linear regression model, supporting the hypothesis that culture toward primary care within an institution can influence graduates’ primary care interest.
Conclusions: Negatively perceived primary care comments were associated with lower interest in primary care careers by medical students. The study provides real-time quantitative data supporting the association between feedback received about primary care and students’ career choices.
(Fam Med 2016;48(1):21-9.)
The United States faces a severe primary care physician shortage.1-4 Despite an increase in the number of medical students over the last decade and a modest increase in those choosing primary care careers,5 the United States is not projected to meet its national primary care needs.6-8 Although many solutions are needed to address this problem, medical schools partially bear responsibility to increase student interest in primary care careers.
During training, medical students are frequently asked about specialty career preferences, often receiving both solicited and unsolicited feedback on this complex decision. Students may lose interest in primary care because of critical comments from mentors, peers, and educators.9-12 The hidden curriculum—a set of influences that function at the level of organizational structure and culture13—is believed to influence students’ ultimate career choices even more than formal policies or curricula.14,15 Student interest in primary care begins to decrease soon after matriculation to medical school,16-18 and the choice of a career in primary care may be particularly affected by “badmouthing” of the professions by attending physicians.11,19,20 Surveys of medical students have shown a high reported incidence of negative comments about primary care.19,21-23 Although several compelling studies have evaluated the content of these comments, all have relied on student recall for data collection and students’ own reports of how these comments influenced their career choices.21,24
This study aimed to quantify the number and positive-to-negative balance of comments about primary care heard by medical students, using a novel methodology consisting of short message service (SMS) text messaging, sent from mobile telephones at point-of-contact. Text messaging for data collection has been used in several research contexts and is associated with higher compliance and satisfaction compared to other methods.25,26 However, text message data collection is a novel approach to measurement of reactions from medical students as they move through the educational process. This approach has advantages: participation is simple, and data can be collected immediately, possibly increasing compliance and minimizing recall errors and recall bias.
We also aimed to explore the relationship between the number of comments heard about primary care and students’ commitment to a primary care career. Together these findings will improve our understanding of how the hidden curriculum influences students’ career choices.
This observational cohort study used SMS text messages sent from medical students as the primary data collection method. Medical students from the College of Medicine at SUNY Upstate Medical University (SUNY-Upstate), Michigan State University College of Human Medicine (MSU-CHM), and Wayne State University School of Medicine (WSU-SOM) participated. These schools were chosen primarily for convenience but represent diverse approaches to undergraduate medical education. MSU-CHM is a public, community-based medical school with seven clinical campuses in rural and urban communities across Michigan. Wayne State University, in Detroit, is an urban public institution and the largest single-campus medical school in the nation. SUNY-Upstate in Central New York is a public medical school offering two options for clinical education: a main campus centered around a university hospital and a distal clinical campus with a longitudinal family medicine clerkship. The percentage of 2014 graduates matching into family medicine from each of these three schools ranged from 4.7%–7.9% (2014 national average 8.5%).27
Medical students from all 4 years of training were recruited from the three schools using e-mail invitations, posters, and announcements. The recruitment letter described the project as a study of feedback that medical students receive about pursuing a career in primary care. All enrolled medical students, regardless of interest in primary care, were invited to participate. Using the random number generator in Microsoft Excel 2007, interested students were randomized into three cohorts, which allowed for data collection over a longer time period. Post-randomization adjustments balanced gender, medical school, and class year distribution.
Each student’s cellular telephone number was stored as a data identifier, linking survey responses and text messaging data. Participants’ names were never linked with mobile numbers. SurveyMonkey.comTM was used to distribute survey instruments to participants via email.28
During each 30-day data collection period, students were instructed to text only a single character, either a “+” or a “-”, to a dedicated study phone whenever they heard a positive (+) or negative (-) comment about primary care from an attending or resident physician. Students were encouraged to consider the tone of the statement, as well as the literal content, to allow for correct classification of “backhanded compliments” and sarcasm and to capture students’ individual interpretations of comments. Investigators specifically did not instruct students in what would constitute positive or negative statements, as the purpose was to capture only the student’s impression of a comment. We felt this would better represent the true effect of any comment, regardless of intent. We deliberately limited data to a single character to allow students a method that was quick and unobtrusive—with the goal of maximizing compliance with the data collection method. At the start of data collection and weekly throughout the data collection period, students received a text asking them to respond with the word “test” if participating in the study. The purpose of these “test texts” was to document continued participation even if students heard no relevant comments.
Students who completed all phases of the study received $50. Participants who completed only one phase received $25. The study protocol was approved by the Institutional Review Boards at each participating university.
We surveyed study participants before and after each 30-day texting period. The survey instruments were tested during a pilot study conducted at Wayne State University in 2010 and modified for use at multiple institutions across two time periods. The content of each of the four surveys is detailed in Table 1, and instruments are available in the STFM Resource Library.29
Each pre-texting survey (Surveys 1 and 3) asked about clinical or preclinical assignments in the next 30 days. The initial pre-texting period survey (Survey 1) also collected demographic information (age, gender, race, and ethnicity), rural, suburban, or urban origins, and attitudes toward primary care. Attitudes toward primary care were assessed using a scale developed at MSU-CHM and abbreviated for use in our study. This scale was developed using a comprehensive literature review and has been positively correlated with students’ experience on a family medicine clerkship in a small pilot study.30
Students were re-surveyed after each texting period (Surveys 2 and 4). At each post-texting survey, students were given a list of comments about primary care careers that had been reported previously in the literature21-23 and asked whether they heard similar comments during the preceding texting period. Students could also free-text other recalled comments. In addition, at each survey administration (Surveys 1–4), students were asked, “Will you choose a primary care specialty (such as family medicine, general internal medicine, or general pediatrics) for your career?” This question was rated on a 5-point scale (Yes, Probably, Maybe, Probably Not, No).
To simplify subsequent surveys, correlational analyses were used to compare Likert-scaled questions pertaining to beliefs about primary care with the statement “Will you choose a primary care specialty (such as family medicine, general internal medicine, or general pediatrics) for your career?” We decided a priori that if the majority of these questions were significantly correlated (P<.05), then the matrix of questions could be eliminated from the second-round post-survey. The majority of questions pertaining to beliefs about primary care did have a moderate-to-strong and statistically significant correlation with the career statement. Thus, all primary care attitude questions except the career choice question were eliminated from subsequent surveys.
To evaluate reliability and satisfaction with the data collection process, at the completion of each 30-day texting period we assessed respondents’ self-reported compliance with data collection and satisfaction with texting as a method (Surveys 2 and 4). In order to evaluate the degree of recall bias, at the completion of the study we asked students whether they recalled more positive or negative comments about primary care (Survey 4).
Text data were compiled to create a count of total negative and positive comments reported by all participants and a count of negative and positive comments reported by each participant. An individual negative-to-positive ratio was calculated for each participant by taking the participant’s total count across both periods of all negative and positive comments and dividing total negative count by positive count.
Linear regression was used to predict post-study primary care interest. The raw totals of comments per individual, and the individual negative-positive comment ratios, were modeled as predictors of respondents’ self-reported likelihood of choosing a primary care career from the final survey. The change in student attitude toward primary care that was seen with each variable or group entered in the model was reported as the difference from the baseline mean (the z-normalized value) in order to simplify comparisons between groups. Models were estimated using ordinary least squares (OLS) multiple linear regression techniques with assessment of standard OLS assumptions. We incorporated race, ethnicity, gender, institution, information about family origins, year in medical school, and baseline attitudes toward primary care as covariates, to account for other factors that may impact primary care career interest. Covariates were first analyzed for correlation with one another and with the outcome variable and then entered iteratively in groups, in forward stepwise fashion, with each successive model presented. The coding for the variables in the model is detailed in STFM’s Resource Library.31 Analyses were conducted using SPSS v.20.
A total of 139 students participated in at least one aspect of the study (one survey or text messaging data collection period). Of these, 120 (86.3%) participated in at least one texting phase, 88 (63.3%) participated in both texting phases, and 87 (62.6%) participated in all aspects of study (all surveys and both texting periods). The demographics of fully participating students are described in Table 2. At the start of the study, 37% of responding participants indicated an interest in primary care; 32% responded they probably or definitely would not choose a primary care career, and 31% indicated they were not sure. Breakdown of primary care interest by school is outlined in Table 3.
The sum of positive comments across all three institutions (851) was higher than the sum of negative comments (616) for the 120 students who participated in either texting period. As Table 4 indicates, when aggregated, the mean number of positive comments was also greater than or equal to the mean number of negative comments, at all three institutions (ratio negative-to-positive comments, in aggregate=0.72). However, when ratios were reported and analyzed individually by student, the mean ratio per student of negative-to-positive comments for two institutions was above 1.00. We believed this individual student ratio best represented individual student experiences and therefore used the individual student ratio, as well as total number of comments per student, in the final OLS models. Standard deviations for number of negative comments, number of positive comments, and ratio of comments were large.
OLS regression modeling was initially performed for both positive and negative comments and ratios. In simple and limited linear regression models, the raw number of positive comments had no statistically significant association with students’ interest in primary care at the end of the study period. Because of this, we included only the number of negative comments and negative-to-positive ratio of comments, omitting positive comments from the analysis.
In the initial, uncontrolled OLS regression model (Table 5), an increasing number of negative responses was significantly associated with a lower level of interest in primary care at the completion of the study period (β=-.04, P=.014). The association between an increasing ratio of negative-to-positive comments and lower interest in primary care initially approached but did not reach significance (β=-.145, P=.084).
We then incorporated race, ethnicity, rural upbringing, gender, institution, family information, and baseline attitudes toward primary care, iteratively, to account for other factors that may impact primary care career interest. The ratio of negative-to-positive comments became significant (β=-.191, P=.028) with the addition of the variables controlling for individual characteristics (race, gender, rural upbringing) and institution. When the pre-participation interest in primary care was added to the model, all other covariates lost statistical significance, indicating this variable had a stronger influence than all other variables. Year in medical school and ethnicity were not significant in a priori testing and were excluded from the analysis.
To summarize, the total number of negative comments per student had a small but significant correlation with a decrease in likelihood of choosing a primary care career of 0.04 (on a 5-point Likert scale) with each negative comment. The correlation with the ratio of negative-to-positive comments per student was larger (-0.145 on a 5-point Likert scale) but did not reach significance until the effects of other factors, including institution, were included in the analysis. However, during the study period, neither effect was enough to overcome the large influence of a student’s initial impression of primary care.
Although all common primary care comments included in surveys (Table 1) were noted by at least some of our participants, three comments were noted by the majority of participants. Over half of participants reported hearing “Primary care physicians have better relationships with patients” (127/222, 57%), “Primary care physicians are needed in rural areas” (138/222, 62%), and “Primary care income is low” (151/222, 68%). At the conclusion of the study, the majority of students recalled hearing more positive comments about primary care during the texting periods than negative comments (more positive 42/102, 41%; more negative 27/102, 26%; about equal 33/102, 32%).
Of the total valid sample, 74.3% indicated they submitted a text every time or more than half the time they heard a comment, similar to previously published compliance with texting as a data collection method.26 Over 90% of respondents thought text messaging was a good method of data collection.
Attaining and retaining student interest in primary care is essential to meet the health care needs of the nation.32 Although the medical school environment has been identified as a factor influencing specialty choice,15,33 we are not aware of any studies using point-of-contact quantitative data to describe the messages students report hearing about primary care. In addition, this work begins to explore the association of such comments with students’ primary care career interest.
Most published literature describing the hidden curriculum has reported a negative bias toward primary care or generalist careers, with students retrospectively recalling more negative input than positive.19,21,22,24 Interestingly, in our study, students in aggregate reported and recalled more positive comments than negative over the study period. However, when the mean ratio of negative-to-positive comments was analyzed per student, more students reported a negative balance of comments, indicating that the excess of positive comments in the aggregate tally was due to an outlier effect. This individual ratio better represents the impact of comments on each student and supports previous studies that have reported more negative input received by students regarding primary care careers.
Our results suggest that negative comments about primary care are associated with lower student interest in primary care. Notably, the association between an individual student’s ratio of negative-to-positive comments and lower interest in primary care reached statistical significance in this analysis only after institution was included. This important finding supports the hypothesis that the culture toward primary care within an institution can influence graduates’ primary care interest.
All of the frequently identified comments described in previous studies were noted by some of our participants, suggesting that these common beliefs about primary care remain entrenched within the medical education culture. In this study, financial concerns were discussed most often as reasons why students should not choose primary care careers. Despite evidence that primary care careers will be financially sustainable for most medical students,34 negative attitudes about primary care physician income pervade the medical culture.35
This study has several limitations. We collected data over a short time period: each data collection lasted only 1 month, and students participating in both data collection periods were enrolled for 6 months. Although we detected an association between more negative comments and declining interest in primary care, the study duration may not have been long enough to overcome the pre-test inclination toward a primary care career, which was the strongest predictor of posttest interest. In this relatively small sample, it is possible that smaller effects of other predictors were also masked by this large effect. Future studies would be useful to follow students longitudinally throughout medical school, measuring the impact of positive and negative comments at different time points.
The study was also limited by the inclusion of only three US allopathic medical schools, two within a single state. Although the results reported here suggest that institutional environment does have some effect on students’ intentions, including a broader range of medical schools and analyzing the prevalence of positive and negative comments based on institutional characteristics, such as campus structure, mission, tuition cost, and public funding, would be valuable in a future study. A comparison group would also be useful, as the act of monitoring comments itself may have affected primary care interest. Due to these limitations, this should be considered an exploratory pilot study; future researchers might consider a longer time-frame for data collection, a larger sample, accounting for institutional variables, and use of more institutions with greater diversity. Despite these limitations, our results suggest that accumulated negative comments may negatively influence a student’s primary care intent.
Medical student specialty choice is a complex decision affected by many variables, one of which is the input of educators, mentors, and peers. By presenting point-of-contact data and exploring associations with career choice, this study provides further evidence that the hidden curriculum has a measurable impact on the future physician workforce. As medical schools acknowledge their social responsibility to meet the workforce needs of the nation, the importance of understanding the cultural biases of medical professionals, educators, and institutions and the impact of those biases on the primary care workforce will become paramount. To effectively develop the primary care physician workforce for this country, we need to send a clear message from all levels, individual to institutional.
Acknowledgments: Funding/Support: This study was funded by a $10,000 Intramural Mini Grant (grant # 98745) from William Beaumont Hospital Research Institute (Andrea Wudyka, PI; Elie Mulhem, mentor). This project was also partially supported by the Health Resources and Services Administration (HRSA) of the US Department of Health and Human Services (HHS) under grant D54HP23297, “Administrative Academic Units,” (Christopher P. Morley, PI/PD); total award amount for AY 2014 $154,765; approximately 95% of this project was funded by non-governmental sources). This information or content and conclusions are those of the author and should not be construed as the official position or policy of, nor should any endorsements be inferred by HRSA, HHS, or the US Government.
Presentations: This study has been presented in various forms at the following national conferences: 2014 Annual North American Primary Care Research Group (NAPCRG) Meeting, New York City, NY; 2014 Society of Teachers of Family Medicine (STFM) Conference on Medical Student Education, Nashville, TN (conference abstract published in Teaching and Learning in Medicine (Melissa Robinson and Michael D. Mendoza. Abstracts from the proceedings of the 2014 STFM Conference on Medical Student Education. Teaching and Learning in Medicine 2014;27(2): 226-232); 2013 Annual NAPCRG Meeting, Ottawa, Ontario; 2013 STFM Conference on Medical Student Education, San Antonio, TX; 2012 Annual NAPCRG Meeting, New Orleans, LA; 2011 NAPCRG Meeting, Banff, Alberta; and 2011 STFM Annual Spring Conference, New Orleans, LA.
Corresponding Author: Address correspondence to Dr Wendling, Michigan State University, Department of Family Medicine, 11133 Summerhill Way, Charlevoix, MI 49720. 231 675-2245. Fax: 231-582-5338. email@example.com.
From the Department of Family Medicine, Michigan State University College of Human Medicine (Drs Wendling and Phillips); Department of Family Medicine and Community Health, Oakland University William Beauont School of Medicine (Drs Wudyka and Mulhem); Department of Internal Medicine (Dr Levine) and Department of Family Medicine and Public Health Sciences (Dr Neale), Wayne State University; and Departments of Family Medicine, Public Health & Preventive Medicine, and Psychiatry & Behavioral Sciences, Upstate Medical University, State University of New York (Dr Morley).
A National Survey of Research Requirements for Family Practice Residents and Faculty
Anne Victoria Neale
Fam Med 2002;34(4):262-267
A Qualitative Study of Medical Students in a Rural Track: Views on Eventual Rural Practice
Carrie Roseamelia, MA; James L. Greenwald, MD; Tiffany Bush, MS; Morgan Pratte; Jessica Wilcox, MS; Christopher P. Morley, PhD, MA, CAS
Fam Med 2014;46(4):259-266
A Step Ahead—Evaluating the Clinical Judgment Skills of Incoming Interns
Andrea Wendling; Philip Baty
Fam Med 2009;41(2):111-115
A Systematic Approach to Diabetes Mellitus Care in Underserved Populations: Improving Care of Minority and Homeless Persons
Philip J. Baty; Susan K. Viviano; M. Rosita Schiller; Andrea L. Wendling
Fam Med 2010;42(9):623-627
An Intern's Education
Fam Med 2005;37(1):15-15
Answering Funding Preferences for Medical School Graduates in Underserved Areas
Anne Victoria Neale; David L. Gaspar; Kurt Metzger
Fam Med 2000;32(7):456-458
Assessing Resident Competency in an Outpatient Setting
Andrea L. Wendling
Fam Med 2004;36(3):178-184
Before the Resurrection
Julie Phillips, MD, MPH
Fam Med 2016;48(10):818-818
Fam Med 2015;47(7):564-564
Andrea Wendling, MD
Fam Med 2016;48(5):341-342
Dedicated Issue on Rural Health: Inspiration, Celebration, and a Challenge for the Future
Fam Med 2010;42(10):690-692
Educational Debt and Career Choice: Every Student Matters
Fam Med 2013;45(7):516-516
Expectations of the Competencies of Entering Family Medicine Residents: Do Members of the Family Agree? A CERA Study
Sarah J. Swofford, MD, MSPH; Amanda L. Allmon, MD; Christopher P. Morley, PhD; Robin L. Kruse, PhD; Elizabeth Garrett, MD, MSPH
Fam Med 2016;48(8):613-617
Family Medicine or Primary Care Residency Selection: Effects of Family Medicine Interest Groups, MD/MPH Dual Degrees, and Rural Medical Education
Elizabeth Wei McIntosh; Christopher P. Morley, PhD, MA
Fam Med 2016;48(5):385-388
How Do Medical Students View the Work Life of Primary Care and Specialty Physicians?
Julie Phillips; David Weismantel; Katherine Gold; Thomas Schwenk
Fam Med 2012;44(1):7-13
Insights From Practice-based Researchers to Develop Family Medicine Faculty as Scholars
Linda M. Roth; Anne Victoria Neale; Kambria Kennedy; Mark J. DeHaven
Fam Med 2007;39(7):504-509
Interactive Peer Review: An Innovative Resident Evaluation Tool
Andrea Wendling; Lisa Hoekstra
Fam Med 2002;34(10):738-743
Keep on Driving
Fam Med 2016;48(10):817-817
Making Research Meaningful in a World With Limited Resources
Fam Med 2013;45(9):613-614
Medical Student Debt and Primary Care Specialty Intentions
Julie P. Phillips; David P. Weismantel; Katherine J. Gold; Thomas L. Schwenk
Fam Med 2010;42(9):616-622
Needing a Nudge: The Effect of Encouragement on Submission Rates and Journal Selection
Andrea Wendling, MD; Joseph Brocato, PhD
Fam Med 2014;46(6):454-458
Practice Management Residency Curricula: A Systematic Literature Review
David E. Kolva; Kathleen A. Barzee; Christopher P. Morley
Fam Med 2009;41(6):411-419
Andrea L. Wendling, MD
Fam Med 2016;48(8):593-595
STFM 2005 Poetry and Prose Contest Winners
Johanna Shapiro; Andrea Wendling; Emily Ferrara; Kenneth Kessel; Deborah Kasman
Fam Med 2005;37(9):627-628
Fam Med 2016;48(10):816-816
Surviving Your Doctors: Why the Medical System Is So Dangerous to Your Halth and How to Get Through It Alive
Fam Med 2010;42(9):664-664
Teaching Cross-Cultural Communication Skills Online: A Multi-Method Evaluation
Amy L. Lee, MD; Emily M. Mader, MPH, MPP; Christopher P. Morley, PhD, MA, CAS
Fam Med 2015;47(4):302-308
The Effect of a Didactic
Lecture on Resident
Andrea Wendling; Bridget Green
Fam Med 2005;37(6):386-387
The Extent and Methods of Public Health Instruction in Family Medicine Clerkships
Jacob Prunuske, MD, MSPH; Linda Chang, PharmD, MPH, BCBH; Ranit Mishori, MD, MHS; Alison Dobbie, MD; Christopher P. Morley, PhD, MA, CAS
Fam Med 2014;46(7):544-548
The Impact of Administrative Academic Units (AAU) Grants on the Family Medicine Research Enterprise in the United States
Christopher P. Morley, PhD, MA; Brianna J. Cameron, MPH; Andrew W. Bazemore, MD, MPH
Fam Med 2016;48(6):452-458
The Social Mission in Medical School Mission Statements: Associations With Graduate Outcomes
Christopher P. Morley, PhD, MA; Emily M. Mader, MPH, MPP; Timothy Smilnak; Andrew Bazemore, MD, MPH; Stephen Petterson, PhD; José E. Rodríguez, MD; Kendall M. Campbell, MD
Fam Med 2015;47(6):427-434
The Story of Trauma
Fam Med 2015;47(1):5-6
Third- and Fourth-Year Medical Students’ Changing Views of Family Medicine
Julie Phillips, MD, MPH; Iain Charnley, MD
Fam Med 2016;48(1):54-60
Use of On-line Evidence-based Resources at the Point of Care
Kendra Schwartz; Justin Northrup; Nejla Israel; Karen Crowell; Nehman Lauder; Anne Victoria Neale
Fam Med 2003;35(4):251-256
Using the Guidelines for Adolescent Preventive Services to Estimate Adolescent Depressive Symptoms in School-based Health Centers
Michael T. Kopec; Jaclyn Randel; Batool Naz; Monina Bartoces; Joseph Monsur; Anne Victoria Neale; Kendra L. Schwartz
Fam Med 2010;42(3):193-201
Fam Med 2014;46(7):556-557
What Authors Need to Know About the Way Editors Think: Report From the Congress on Peer Review in Biomedical Publication
Fam Med 2002;34(2):138-139
Leave a comment on this article
You must complete all fields to comment