Response to Jacob Schor


Benton Bramwell, ND, and Matt Warnock

We are grateful that Dr. Schor has taken the time to read our article and comment upon a reference within it.  We welcome the discussion and are glad it is taking place.  We have no quarrel with the probabilistic arguments referenced by Dr. Schor in his comment, which are put forward by Senn.1  As we understand it, a main point of those arguments is that the process of analysis in RCTs produces meaningful intervals around the point estimate that need to be used to interpret estimates correctly and that the uncertainty  captured within those confidence intervals is expected to be larger with any variance caused by imbalance.

Having acknowledged that, we wish to speak briefly to the process of clinicians utilizing information about probabilities to make decisions in a clinical setting.  In a very real sense, the clinician is the client of the professional researcher.  As the client, the clinician’s decision-making needs should be optimally supported by the researcher. This context leads to an interesting question for clinicians who interpret point estimates and the intervals around them in the process of clinical decision-making.  When interpreting data as to the potential impact of an intervention, both estimate and associated intervals, would the clinician-client want these values to represent information of the highest precision and the least bias possible?

If the answer to that question is yes, then where data from RCTs is relied upon we feel there is value in having as much balance of potentially confounding variables as possible.  As Nguyen discusses in another publication,2 benefits of balance include an estimate of treatment effect that is closer to the true value and confidence intervals that are conservative (that is, variance is overestimated).  A like-for-like comparison makes it easier for the client, the clinician, to make better decisions for patients when data from RCTs are relied upon in the decision-making process, from our point of view.

While a commonly expressed hope of simple randomization is that the process will lead to an even distribution, a balance of variables between groups,3,4 Nguyen’s work helps us to understand that reliably removing random differences between groups with simple randomization is going to be very difficult in small or medium size studies.5  Therefore it probably does make sense to see randomization’s role as one of helping to manage uncertainty.  To the point we make in our article, this further illustrates that there are limitations as to what simple randomization can accomplish.  Further reading on the topic of randomization’s limitations, including perhaps even a potential for it to introduce confounding into smaller studies, is provided by Saint-Mont.6

We are concerned that many tend to blindly and categorically consider significant findings from RCTs, particularly those from randomized, double-blind, placebo-controlled trials, to be “approved” for clinical decision-making, while data from other sources are considered categorically to be “not approved.” Given the limitations of simple randomization, the frequent lack of effective blinding, and ethical drawbacks of placebo use, we feel firmly that helpful data from other sources, such as can be obtained from large prospective cohort-based observation studies, should be more widely regarded and utilized. 

References

  1. Senn S. Empirical studies of balance do not justify a requirement for 1,000 patients per trial. J Clin Epidemiol. 2022 Aug;148:184-188. doi: 10.1016/j.jclinepi.2022.02.010. Epub 2022 Mar 4. PMID: 35248697.
  2. Nguyen TL, Xie L. Incomparability of treatment groups is often blindly ignored in randomised controlled trials – a post hoc analysis of baseline characteristic tables. J Clin Epidemiol. 2021 Feb;130:161-168. doi: 10.1016/j.jclinepi.2020.10.012. Epub 2020 Oct 17. PMID: 33080343.
  3. Kendall JM. Designing a research project: randomised controlled trials and their principles. Emerg Med J. 2003 Mar;20(2):164-8. doi: 10.1136/emj.20.2.164. PMID: 12642531; PMCID: PMC1726034.
  4. Suresh K. An overview of randomization techniques: An unbiased assessment of outcome in clinical research. J Hum Reprod Sci. 2011 Jan;4(1):8-11. doi: 10.4103/0974-1208.82352. Retraction in: J Hum Reprod Sci. 2023 JanMar;16(1):87. PMID: 21772732; PMCID: PMC3136079.
  5. Nguyen TL, Collins GS, Lamy A, Devereaux PJ, Daurès JP, Landais P, Le Manach Y. Simple randomization did not protect against bias in smaller trials. J Clin Epidemiol. 2017 Apr;84:105-113. doi: 10.1016/j.jclinepi.2017.02.010. Epub 2017 Feb 28. PMID: 28257927.
  6. Saint-Mont U. Randomization Does Not Help Much, Comparability Does. PLoS One. 2015 Jul 20;10(7):e0132102. doi: 10.1371/journal.pone.0132102. PMID: 26193621; PMCID: PMC4507867.

Published December 2, 2023

About the Authors

Benton Bramwell, ND, is a 2002 graduate of National College of Naturopathic Medicine who practiced primarily in Utah while helping to expand the prescriptive rights of naturopathic physicians in that state.  Currently, he owns and operates Bramwell Partners, LLC, providing scientific and regulatory consulting services to both dietary supplement and conventional food companies.  He and his wife, Nanette, have six children and two grandchildren; they live in Manti, Utah.

Matt Warnock is an accidental herbalist, who received his MBA and Juris Doctor from BYU, then worked as an attorney, litigator, and business consultant until 2000. He then joined RidgeCrest Herbals, a family business started by his father, and started learning about
herbal medicine, focusing especially on complex herbal formulas. He has two U.S. patents for herbal formulations and methods. He lives near Salt Lake City with his wife, Carol; they are the parents of three children and four grandchildren.