By Joshua Z. Goldenberg, ND

Warning, this article is going to be about evidence-based medicine (EBM). Check in with yourself. What are you feeling?

Joking aside, while EBM has become established over the past 20 years in conventional and integrative medicine environments, it is often still met with skepticism (Trinder 2000). Let’s explore that.

In the aptly titled BMJ paper “EBM what it is and what it isn’t,” David Sackett, arguably the father of EBM, defined it as the “conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients’’ further delineated as a confluence of physician experience, patient values, and best evidence (Sackett 1996).

The first two elements don’t ruffle too many feathers. Physician experience comes from just that, experience. Being respectful of and working with patient values is something I believe naturopaths specifically excel at, so no problem there. Rather it is the third “best evidence” element that I often see my students (ND or MD) struggle with. The reason for this struggle, I believe, comes down to two issues regarding medical research: there is too much of it and most of it isn’t very good.

In this essay I would like to discuss these two issues, suggest what we as clinicians can do about them, and argue why this is particularly important for naturopathic physicians.

Let us begin with the glut of medical research. There is a tome of research articles out there, and they are increasing at an exponential rate. Currently, Pubmed alone has more than 26 million citations. Gone are the days when you could get your JAMA in the mail each week, flip through it on the weekend, and feel up-to-date with current medical thinking.

And what about the quality of the deluge? Unfortunately, we can take little solace here either. Most research is of low methodological quality and likely reaches false conclusions and even the papers that look good initially should be suspect (Ioannidis 2005). Why? While there are many reasons, one is certainly because the classic EBM hierarchy, often depicted as a pyramid with systematic reviews and randomized controlled trials (RCTs) on top, has been “hijacked,” to quote John Ioannidis, a leading EBM proponent. He argues that vested interests have taken advantage of this hierarchy and often produce papers that appear to meet criteria for ‘top of the pyramid’ evidence but actually have a significant risk of bias which often requires some digging to uncover (Ioannidis 2016). Examples would include publication bias, weak comparator choice, selective outcome reporting, and spin in the abstract and discussion sections.

For these reasons, the overworked and time-poor physician has had to shift in practice to pre-appraised evidence sources such as UpToDate, Dynamed, practice guidelines, and others (Chapa 2013). While there are some issues with relying on pre-appraised evidence and some have argued this may actually undermine the anti-authoritarian nature of EBM to begin with (Goldenberg 2009), it is likely net positive and the time-saving nature of it is essential to any time-crunched physician.

Now comes the point where I tell you why we as naturopaths, in particular, need to be rockstars. While naturopathic researchers are toiling away every day establishing an evidence base for the profession, to date, naturopathic medicine has a relative dearth of high-quality clinical evidence. This thin evidence base makes evidence-based guidelines difficult to construct.

So, what is an evidence-based naturopath to do? Well, pragmatically this means that we tend to go directly to whatever primary research sources there are. Unfortunately, these often tend to be methodologically weaker and are of course rarely pre-appraised by a methodologist.

For this reason, I think we need to be our own methodologists. I know we went to naturopathic medical school to be clinicians, but I believe that modern medicine needs clinicians to also be partial scientists. We need to be able to identify relevant evidence, ruthlessly assess it, and apply it to our patients. But how do we do a good job at ferreting out bias and presenting only the most trustworthy material to our patients?

I have taught critical evaluation of the medical literature to NDs, MDs, DOs, chiropractors, medical students, and others. I have taught it online and in person. I have lectured in front of large classes and guided small journal clubs. I have taught day-long boot-camps and multiple-quarter-long training programs. So believe me when I say there are MANY tips and techniques you can learn to master evidence evaluation. However, you don’t need to be a master to start, but you do need to start somewhere.

Like our medical practices themselves, critical evaluation of the medical literature takes practice. While we left naturopathic medical school competent, we certainly didn’t leave as experts. That’s what a life of practice is all about: honing, refining, and becoming the best physician you can be. It is the same with research evaluation. Take a page out of Dr. Bastyr’s book and critically read an article every night, or at least once a week. Excellence will come with time and practice.

So with a bit of grace for yourself and the learning process, I would suggest that you start with three questions whenever you read a research paper.

Question number one. Can you trust it? This speaks to the risk of bias of the study. Some of these elements you likely already know such as checking for blinding, but there are many more domains and in depth tools you could use. I would recommend learning the Cochrane Risk of Bias Tool (Higgins 2011) if you wanted to do a deep dive, but the first thing I would look for is conflict of interest. Who are the authors of the paper? Who was the sponsor? Try to understand their motivations. Be skeptical. A quick scan of the authors and their affiliated institutions can be a tip off. Is one of the authors an employee of the company whose natural product is being investigated?

Question number two. How big is it? We often get so caught up in p values. While certainly important, p values only tell you about statistical significance. They tell you nothing about clinical significance. We need to be better at understanding the effect sizes of interventions. This could be described in risk ratios, odds ratios, numbers needed to treat, standardized mean differences, and more. While often confusing, try to get a sense of what the actual effect size is. Ask yourself, is this an important difference? Would this matter to my patient? For example, consider a study that found a statistically significant effect of supplement X on blood sugar but the mean treatment group had a fasting glucose of 188 post-treatment versus the control group’s 189. Would that statistically significant difference be significant to your patient?

Finally, question three. Does it apply to my patient? If the study passes the ‘trustiness test,’ and clinical relevance check, we need to then ask if it is applicable to our patient. For this I often advise my students to look at the baseline data table for the study participants. Are all the participants normotensive, normal weight, and of Asian descent while your patient is hypertensive, overweight and Caucasian? Is the analysis based on per protocol compliance but your patient at best does half the things you recommend? While a research study will rarely match up perfectly with your patient, at least see if you are in the same ball park.

In summary, busy clinicians who strive to practice in an evidence-based manner face the conundrum of a plethora of poor quality evidence from which to extract best evidence. To deal with this, clinicians have moved away from individual article review and towards point of care pre-appraised evidence. Unfortunately naturopaths (and any other medical provider in a field with limited clinical evidence) are forced to source directly from primary literature. This literature is often riddled with bias, much of it subtle and un-appraised. We must never accept the results of a paper at face value but rather be our own methodologists, our own ferreters of bias, our own critical evaluators of the medical literature rockstars.

Dr. Goldberg will be teaching a boot-camp on Evidence-Based Medicine for CEUs at the Bastyr University on February 25, 2017.

Bibliography

Chapa D, Hartung M, Mayberry L, Pintz C. Using preappraised evidence sources to guide practice decisions. Journal Of The American Association Of Nurse Practitioners. 2013; 25(5): p. 234-43.

Goldenberg MJ, Borgerson K, Bluhm R. The Nature of Evidence in Evidence-Based Medicine: Guest editors’ introduction. Perspectives in Biology and Medicine. 2009; 52(2). p 164-7.

Higgins JPT, Green S. (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org.

Ioannidis JPA. Why Most Published Research Findings Are False. PLoS Medicine. 2005; 2(8). e124.

Ioannidis JPA. Evidence-based medicine has been hijacked: a report to David Sackett. Journal of Clinical Epidemiology. 2016; 73. p 82-6.

Sackett D, Rosenberg W, Gray J, Haynes R, Richardson W. Evidence based medicine: what it is and what it isn’t. BMJ. 1996; 312(7023). p. 71-2.

Trinder L, Reynolds S. Evidence-Based Practice: A Critical Appraisal. Oxford: Blackwell Science; 2000. p 249.

ALSO, this page has Goldenberg’s picture and bio (at the bottom): https://bastyr.edu/civicrm/event/info?id=2181&reset=1