Simple Ways to Evaluate the Validity of a Research Study

I know first hand that scientific studies often make claims which are not valid. I know this because I worked as a validation expert (the title is hysterical) as a member of research teams for a major pharmaceutical manufacturer. In this role, I saw statistics manipulated every day to change the conclusions drawn. Typically, statistics were manipulated to make dangerous outcomes appear safe. This practice is mainstream in the pharmaceutical industry. It is an insidious habit, but one that is not limited to the pharmaceutical industry. I’m not the only one to draw this conclusion. If you’d like more information, please read The Mainstream Manufacture of Misinformation.

The bottom line is that you can rarely trust the media’s reports on a study’s findings. Most media professionals don’t have time (or don’t take time) to actually read a study. Instead, they rely on other media reporting agencies to draw conclusions for them. The only true way to evaluate a study is to actually read it or to read a synopsis of it. Specifically, one must check the number of participants studied, the study protocol used, who sponsored the study and paid for it, and other factors that may influence the conclusions. More than once, the media has reported on a study but failed to share comments made by the people who actually performed the study. Instead, media channels reported on conclusions drawn by companies which could potentially be negatively affected by the study. Do your own research. Don’t trust mainstream media.

Before accepting the findings of a research study, one must evaluate the research techniques used, the study protocol, and other factors to determine if the study’s claims are valid. Let’s look at a few items that should be reviewed before accepting the findings of a research study:

  1. Number of Participants: A study using three participants is obviously not going to be as predictive as a study using a much larger group of participants. Yes, I have seen studies PUBLISHED which based a conclusion on three people. In the study I’m referencing, a Japanese study on the herb Lion’s Mane, the three people negatively affected by the herb had cancer and were receiving chemotherapy. The conclusion which was published was that Lion’s Mane unequivocally causes liver damage. In my mind, this study proved nothing, since chemotherapy is known to cause liver damage and the people studied had severely compromised health. When reviewing a study, always check the study protocol to see what size “control” group was used and what “controls” were in place. More on the control group follows.
  • Specific Study Protocol:When reviewing a study done on pharmaceuticals or herbs, very carefully review how the study was performed. Things to check include the dosage used, what effects were studied, how the effects were measured, which part of the herb was used, etc. A 2005 study published in the New England Journal of Medicine (Turner, 2005) claimed the herb echinacea had no effect on the duration or severity of colds. However, the study only used 900 mg of the herb on a daily basis. Herbalists typically recommend a minimum dosage of 2700 mg daily. A study which only used 900 mg was bound to fail. Additionally, this study used extracts that were created in the lab by scientists who knew nothing about how to make herbal extracts and who used chemicals to make the extracts. I’ve been an herbalist for 25 years. Trust me when I say the art and science of creating a valid herbal extract requires extensive training, practice and research. Additionally, an extract made using a chemical such as ethanol (one of the chemicals used in this study) will be contaminated in a manner which could negatively impact the herb’s effectiveness. I shudder to think what the study participants were actually taking.Another problem in this study was that the extracts were made using a single variety of the herb and were made from the herb’s root. A truly effective herbal extract of echinacea will include extracts from the three most common varieties and will include extracts from the roots and aerial portions of the herb. Quite simply, there was no way this study could have shown echinacea to have positive effects. The study protocols were so flawed that it leads one to believe the desired outcome of the study was to prove echinacea useless. This study had multiple other flaws, but I’ve shared the main ones. You can read an abstract of the study yourself here: you’re interested in seeing how one person tore apart a research study and effectively proved it to be invalid, I encourage you to read Kevin Trudeau’s research on one study done on Gardasil:
  • Study Sponsor and Resource Provider: It would be nice to think that all studies are commissioned purely for the sake of science and not because someone has a point to prove. The fact is that many studies are set up and paid for because the sponsor has a desired outcome. While this practice is in direct opposition to the scientific method, corporate greed guarantees that studies are often commissioned whose protocol is designed to have a specific outcome and to prove conclusions which were drawn in advance. I always get a bit leery of any study paid for by a pharmaceutical company (or a research facility highly funded by a pharmaceutical company) which claims to prove an herbal extract is absolutely ineffective. For instance, the Glaucoma Research Foundation sponsored several studies on natural remedies for glaucoma and reported that no natural treatments are effective. I know this is untrue because I reversed my own glaucoma using natural methods, have helped others do the same, and have spoken to natural practitioners who have had similar success with 100s of clients. Anecdotal evidence isn’t scientific, but substantial amounts of it should lead one to desire additional information.

To conclude, don’t assume the reported conclusions of a study are correctly reported and do not accept a study’s validity without digging a bit deeper. Being able to separate the wheat from the chaff when reviewing scientific evidence can help avoid being deceived.

The following two tabs change content below.
Dr. Pamela Reilly is a Naturopathic Physician dedicated to helping people improve their health and eliminate symptoms using natural, integrative methods. She has over 25 years of experience and has helped men, women and children improve their health using a holistic, client-centered focus. She sees clients in Indianapolis, does house calls, and also conducts consultations via Skype or telephone. Please feel free to contact her or visit her Consultations page for more information. Dr. Pamela speaks nationwide on a wide variety of health topics and welcomes speaking invitations.

Latest posts by Dr. Pamela Reilly (see all)

7 Responses to Simple Ways to Evaluate the Validity of a Research Study

  1. Excellent article – thanks so much for putting all of the effort into this to prove your point!!!

  2. […] Sucralose was discovered by a grad student in London in 1976 while working on a study focused on creasing new INSECTICIDES. That’s right. Splenda was “accidentally” discovered as part of a research study focused on creating new insecticides. (On a side note, DDT, a poison now banned, has a very similar chemical structure to Sucralose. Doesn’t that sound delicious?) McNeil Nutritionals, the manufacturer of Splenda, claims that 100s of studies were done on Splenda. The truth is that most of those studies were designed and paid for by McNeil (which puts their validity in question) and that almost all the studies were performed on animals, were very short term, and did not test safety but instead tested secondary effects such as tooth decay. The truth is that only six human studies were conducted, the longest of which was three months. (Most people use Splenda far longer.) For more information on the ways research studies and their results are often faulty, please read my post Simple Ways to Evaluate the Validity of a Research Study. […]

  3. […] study is a company which manufactures and sells green coffee bean extract. (Please read my article, Simple Ways to Evaluate the Validity of a Research Study for more information on ways to know whether published conclusions are valid or […]

  4. Pamela Reilly, Naturopath, CNHP, CNC, CPH says:

    Thanks, Deb! I think most people assume that scientists MUST be right & don't realize how easy it is to evaluate the protocols to determine if the conclusions drawn were valid. Thanks for your comment. Have a great night!

  5. Deb Willbefree says:

    I appreciate this post. It helps people to begin to look at the details of a research project thoughtfully…almost like a detective looks at evidence.

    I've often read or heard research conclusions and have had the thought, "Wait. what?"

    I finally appreciate all of those research courses that were forced on my in undergrad, then graduate, studies. :} Who knew?


Leave a Reply

Your email address will not be published. Required fields are marked *