Why I Am Believing More and More Nutrition Research Less and Less

By Alan R. Gaby, MD

During the past 46 years, I have reviewed and analyzed more than 50,000 papers from the biomedical literature, most of which were related to the field of nutritional medicine. Doing this work has given me some understanding of how to assess the reliability of a study. Over the past 5 to 10 years, an uncomfortably large and growing number of published papers related to my area of expertise have left me wondering whether the research was fabricated; that is, whether people were writing papers about research that had not actually been conducted. This article should not be construed as an accusation against anyone. But when I see my colleagues citing and relying on research that I find concerning, that concerns me as well.

The studies that have raised my eyebrows have come primarily from Iran and to a lesser extent from China, Egypt, India, Japan, and a few other countries. Characteristics of these eyebrow-raising studies typically include one or more of the following:

1. The study comes from an investigator or research group that has published an enormous number of randomized controlled trials in a relatively short period of time.

2. The number of participants in the trial is unusually large, when considering the resources that appear to be available to the researchers.

3. The recruitment period for the trial is unusually short.

4. The paper is submitted to the journal unusually rapidly after the study is completed, or in some cases before it would have been possible to have completed the trial.

5. A randomized controlled trial is conducted before there is any preliminary evidence of efficacy in humans (such as case reports or uncontrolled trials). Because randomized controlled trials are expensive to conduct, such trials are generally reserved for treatments for which there is some evidence of efficacy.

6. The magnitude of the reported improvement is much larger than is typically seen in trials using just one or two nutrients.

7. No funding source is listed, or the study is listed as self-funded. This issue is especially noteworthy when the number of participants or the number of tests conducted is large, or when participants undergo relatively expensive testing.

8. The design of the study raises ethical issues, such as participants not being permitted to use treatments that have previously been reported to prevent adverse outcomes in the offspring of pregnant women.

9. One or more baseline characteristics of the study group appear to be implausible.

A few of the many papers that have caught my eye are examined below.

Prolific Iranian Researcher

In a double-blind trial from Iran, conducted by Zatollah Asemi and a coworker, and published in the Journal of Clinical Endocrinology and Metabolism, administration of 50 mg of soy isoflavones per day for 12 weeks significantly improved insulin resistance, hormonal status, triglyceride levels, and biomarkers of oxidative stress.¹

For several years I have been watching the work of Asemi with skepticism, in part because of the large number of randomized controlled trials in which he has been involved and the fact that nearly every trial reported positive results (often with a large effects size), regardless of the nutrient being tested or the condition being treated. A researcher might be considered prolific if he or she was involved in, say, half-a-dozen or so randomized controlled trials over a period of a decade. In contrast, Asemi and coworkers have published (by my count) a total of 191 randomized controlled trials, including 148 between January 1, 2016 and March 30, 2019.

In a letter to the Journal of Clinical Endocrinology and Metabolism² regarding the study cited above, I pointed out that, given the stated recruitment period and duration of the intervention, the earliest date the authors would have been able to start evaluating their data was four weeks after the date the paper was submitted to the journal. In the letter I also cited another study³ in which Asemi et al submitted a paper before it was apparently possible to have completed the trial (I have now found 5 such instances of seemingly impossible dates among papers by Asemi et al).

In their response to my letter,⁴ Asemi and his coworker changed the recruitments dates that had been listed in those two papers. One of the changes was to shorten the recruitment period in a gestational diabetes study from two or three months originally, down to three weeks. That is, during a three-week period, they recruited 60 women from a single clinic who were between 24 and 28 weeks’ gestation and who had gestational diabetes. After reading their response, I found data on the birth rate in the region of Iran where the study took place. I also found data (from some of Asemi’s other studies) about the proportion of pregnant women in that region who develop gestational diabetes. Based on that information I calculated that, even if the study had enrolled every woman with gestational diabetes in the entire city where the study was conducted (population, approximately 500,000), no more than 36 women could have been enrolled in a three-week period.

Ethical issues

By my count, Asemi and coworkers have published at least 22 randomized controlled trials on various nutritional treatments for gestational diabetes. One of the earliest studies in that series found that, in women with gestational diabetes, supplementation with magnesium significantly decreased the incidence of neonatal hyperbilirubinemia and infant hospitalizations.⁵ The effect size for both of those outcomes was relatively large, with a number-needed-to-treat of five. And yet, in most of the subsequent studies by Asemi et al that examined a wide range of individual nutrients or a combination of two nutrients, women with gestational diabetes were forbidden from taking any supplements other than iron, folic acid, and the nutrient(s) being tested. One would think that it is unethical for researchers to withhold a treatment (i.e., magnesium) that they, themselves, have previously shown to improve neonatal outcomes.

Vitamin A for Ulcerative Colitis

One hundred fifty Iranian patients (aged 20-45 years; mean disease duration, 4.2 years) who had been referred to a gastroenterology clinic for ulcerative colitis, and who were symptomatic despite treatment with oral and topical 5-aminosalicylic acid, continued their medication and were randomly assigned to receive, in double-blind fashion, 25,000 IU per day of vitamin A or placebo for two months. Mean disease activity, as determined by the Mayo Clinic Disease Activity Index, improved by 22% in the vitamin A group and did not change in the placebo group (p < 0.001 for the difference in the change between groups).⁶

This study (in which Asemi was not involved) raises a number of issues. First, the mean Disease Activity Index at baseline was 8.68 out of a maximum score of 12, indicating that the patients on average had moderate-to-severe disease. One would think that patients referred to a gastroenterologist with moderate-to-severe ulcerative colitis unresponsive to 5-aminosalicylic acid would first be given a trial of a different medication. Instead they were asked to try a relatively low dose of a vitamin, for which there had been no prior clinical evidence of efficacy, and also to accept a 50% chance of being given a placebo. One might question the ethics of conducting such a trial. While all of the participants were said to have given “informed, written consent,” one wonders what information they were provided before giving such consent. Second, the study was presumably quite expensive, since it included, among other tests, 293 endoscopies conducted by an “expert gastroenterologist-endoscopist.” One might wonder how the researchers were able to obtain funding for this study without having first demonstrated possible efficacy with case reports or a small uncontrolled trial. Third, some of the baseline characteristics of the participants are unusual. That is, while the vitamin A group was significantly shorter than the placebo group (mean height, 161.4 cm vs. 165.7 cm; p < 0.001), the mean weight of the vitamin A group was significantly greater than that of the placebo group (73.6 kg vs. 69.8 kg; p = 0.03). The probability that the difference in mean body mass index between groups occurred by chance was less than 1 in 1,000: not impossible, but possibly eyebrow-raising.

Green Tea for Acute Cystitis

Seventy Iranian women experiencing their first episode of acute uncomplicated cystitis (diagnosed by an infectious disease specialist, based on medical history, physical examination, clinical manifestations, and urinalysis) were randomly assigned to receive, in double-blind fashion, 2 g of dried aqueous extract of green tea or placebo once a day in the evening for three days. All patients were treated with trimethoprim-sulfamethoxazole. The green tea extract provided daily 131 mg of (–)-epigallocatechin (EGC), which the authors stated is the only green tea catechin excreted in the urine after oral administration. After three days, the proportion of women who had symptoms of cystitis was 63% in the placebo group and 2% in the green tea group (p = 0.0001). The authors concluded that green tea was an effective adjunct to trimethoprim-sulfamethoxazole in women with acute uncomplicated cystitis.⁷

This study (in which Asemi was not involved) also raises various issues. First, one wonders why an infectious disease specialist would waste his or her time evaluating and treating first episodes of acute uncomplicated cystitis, a condition that is often treated by mid-level practitioners or even over the telephone. Second, only 72% of the women were experiencing cystitis symptoms at the time they were enrolled in the study. That fact might make one wonder why the other 28% would have gone to a medical clinic, and why they were given a diagnosis of cystitis (as opposed to asymptomatic bacteriuria). Third, the rationale for conducting the study is open to question. Based on my review of the research cited in the paper, the lowest concentration of EGC that has been found to inhibit Escherichia coli in vitro is more than 50 times higher than the concentration that would be achieved in the urine after administering the green tea extract used in the study. While a sub-inhibitory concentration of EGC could conceivably act synergistically with antibiotics, the available in vitro data (in the absence of any prior clinical evidence of efficacy) does not seem sufficient to have justified conducting an expensive double-blind trial. Fourth, 91% of the women in the placebo group had bacteriuria at baseline, which suggests that in the vast majority of cases the acute cystitis was due to a bacterial infection. Symptoms of an acute first-episode urinary tract infection typically resolve within 24-48 hours after the start of antibiotic treatment. It is therefore difficult to understand how 63% of the women in the placebo group remained symptomatic 72 hours after they received antibiotics.

Honey for Herpes simplex Gingivostomatitis in Children

One hundred Egyptian children (aged 2-8 years) who presented to the Otolaryngology, Head and Neck Surgery Department, Minia University hospital, Minia, Egypt, with primary herpes simplex gingivostomatitis (PHGS) were randomly assigned to receive, in double-blind fashion, acyclovir suspension plus honey or acyclovir plus placebo. The dosage of honey was 5 ml swallowed slowly over a few minutes, every four hours for seven days. The placebo was a sugary syrup that was similar to honey in consistency and color. All patients underwent viral culture and serology testing for herpes simplex. To be included in the study, patients had to have a positive culture, positive serology, and clinical manifestations of PHGS. The patients were examined at enrollment and on days 3, 5, and 7. The median time until the lesions disappeared was significantly less in the honey group than in the placebo group (3 days [range, 1-7 days] vs. 6 days [range, 3-13 days]; p = 0.02).

This study raises several issues. First, the paper states that there was no funding source, so the study was presumably funded by one or more of the three authors. According to Wikipedia, the monthly salary of doctors in Egypt ranges from $69 to $361, so one wonders how the researchers would have been able to pay for 100 viral cultures and 100 serological tests, as well as the other costs of the study. If the authors did pay for the study, one wonders why they would have been willing to invest in an expensive double-blind trial when there had been no prior clinical evidence that honey is an effective treatment for PHGS. If the patients were being asked to pay for these tests (72% of health care expenditures in Egypt are paid for out of pocket [Wikipedia]), it is likely that many of the parents would have opted for empirical treatment with acyclovir in lieu of laboratory testing.

Second, in order to be included in the study, the children had to have a positive viral culture and positive serology. Herpes simplex cultures take an average of three days to become positive, and it may take up to 14 days for the culture to be reported as negative. Regarding serology tests, it could take weeks or even months after symptom onset for antibodies to appear in the blood. Therefore, it is not clear how the patients could have been enrolled in the study and started on treatment on the day they presented to the clinic. There is no mention of post hoc exclusion of participants after the test results came back.

Third, since the final clinic visit was seven days after treatment was started, and there was no mention of any follow-up after day 7, it is not clear how the researchers could have known that the lesions took as long as 13 days to disappear.

Fourth, some of the baseline characteristics of the children in the study are difficult to comprehend. The mean age was 1.3 years less in the honey group than in the placebo group (5.1 vs. 6.4 years). Yet, despite being younger, the mean body weight was greater in the honey group than in the placebo group (15.6 kg vs. 14.2 kg). It seems highly unusual that 50 five-year-old children would have a mean body weight greater than that of 50 six-year-old children.

The Response from Journal Editors

I have made several attempts to alert journal editors to the types of concerns described in this article. Some editors did not respond to my communication. Others conducted a cursory investigation before dismissing my concerns. For example, one editor forwarded my concern about possible ethics violations to a medical ethicist, who concluded that withholding magnesium from women with gestational diabetes was not unethical because the study had been approved by an ethics committee. One would have thought that, before rendering such an opinion, the ethicist would have been interested in seeing the document the researchers submitted to the ethics committee. Other editors seemed willing to accept a flimsy or factually incorrect response from the study authors and then refused to allow me to submit a follow-up letter. I have been in contact with some other investigators who have attempted to voice concerns about published papers, and their experiences with journal editors are similar to mine.

The Problem Is Getting Worse

During the past few years, the number of “concerning” papers in the field of nutritional medicine seems to have increased exponentially. I spend a lot of time on PubMed, scanning the table of contents of the latest issues of medical journals, looking for research in my areas of interest. I also receive daily emails from PubMed, notifying me about new articles that conform to my various search criteria. I have not attempted to quantify how often I have credibility concerns about new research papers, but it may be as high as a dozen times per month.