Simple Ways to Evaluate the Validity of a Research Study
I know first hand that scientific studies often make claims which are not valid. I know this because I worked as a validation expert (the title is hysterical) as a member of research teams for a major pharmaceutical manufacturer. In this role, I saw statistics manipulated every day to change the conclusions drawn. Typically, statistics were manipulated to make dangerous outcomes appear safe. This practice is mainstream in the pharmaceutical industry. It is an insidious habit, but one that is not limited to the pharmaceutical industry. I’m not the only one to draw this conclusion. If you’d like more information, please read The Mainstream Manufacture of Misinformation.
The bottom line is that you can rarely trust the media’s reports on a study’s findings. Most media professionals don’t have time (or don’t take time) to actually read a study. Instead, they rely on other media reporting agencies to draw conclusions for them. The only true way to evaluate a study is to actually read it or to read a synopsis of it. Specifically, one must check the number of participants studied, the study protocol used, who sponsored the study and paid for it, and other factors that may influence the conclusions. More than once, the media has reported on a study but failed to share comments made by the people who actually performed the study. Instead, media channels reported on conclusions drawn by companies which could potentially be negatively affected by the study. Do your own research. Don’t trust mainstream media.
Before accepting the findings of a research study, one must evaluate the research techniques used, the study protocol, and other factors to determine if the study’s claims are valid. Let’s look at a few items that should be reviewed before accepting the findings of a research study:
- Number of Participants: A study using three participants is obviously not going to be as predictive as a study using a much larger group of participants. Yes, I have seen studies PUBLISHED which based a conclusion on three people. In the study I’m referencing, a Japanese study on the herb Lion’s Mane, the three people negatively affected by the herb had cancer and were receiving chemotherapy. The conclusion which was published was that Lion’s Mane unequivocally causes liver damage. In my mind, this study proved nothing, since chemotherapy is known to cause liver damage and the people studied had severely compromised health. When reviewing a study, always check the study protocol to see what size “control” group was used and what “controls” were in place. More on the control group follows.
- Specific Study Protocol:When reviewing a study done on pharmaceuticals or herbs, very carefully review how the study was performed. Things to check include the dosage used, what effects were studied, how the effects were measured, which part of the herb was used, etc. A 2005 study published in the New England Journal of Medicine (Turner, 2005) claimed the herb echinacea had no effect on the duration or severity of colds. However, the study only used 900 mg of the herb on a daily basis. Herbalists typically recommend a minimum dosage of 2700 mg daily. A study which only used 900 mg was bound to fail. Additionally, this study used extracts that were created in the lab by scientists who knew nothing about how to make herbal extracts and who used chemicals to make the extracts. I’ve been an herbalist for 25 years. Trust me when I say the art and science of creating a valid herbal extract requires extensive training, practice and research. Additionally, an extract made using a chemical such as ethanol (one of the chemicals used in this study) will be contaminated in a manner which could negatively impact the herb’s effectiveness. I shudder to think what the study participants were actually taking.Another problem in this study was that the extracts were made using a single variety of the herb and were made from the herb’s root. A truly effective herbal extract of echinacea will include extracts from the three most common varieties and will include extracts from the roots and aerial portions of the herb. Quite simply, there was no way this study could have shown echinacea to have positive effects. The study protocols were so flawed that it leads one to believe the desired outcome of the study was to prove echinacea useless. This study had multiple other flaws, but I’ve shared the main ones. You can read an abstract of the study yourself here: http://bit.ly/rt1Urc.If you’re interested in seeing how one person tore apart a research study and effectively proved it to be invalid, I encourage you to read Kevin Trudeau’s research on one study done on Gardasil: http://bit.ly/q9Kxxv
- Study Sponsor and Resource Provider: It would be nice to think that all studies are commissioned purely for the sake of science and not because someone has a point to prove. The fact is that many studies are set up and paid for because the sponsor has a desired outcome. While this practice is in direct opposition to the scientific method, corporate greed guarantees that studies are often commissioned whose protocol is designed to have a specific outcome and to prove conclusions which were drawn in advance. I always get a bit leery of any study paid for by a pharmaceutical company (or a research facility highly funded by a pharmaceutical company) which claims to prove an herbal extract is absolutely ineffective. For instance, the Glaucoma Research Foundation sponsored several studies on natural remedies for glaucoma and reported that no natural treatments are effective. I know this is untrue because I reversed my own glaucoma using natural methods, have helped others do the same, and have spoken to natural practitioners who have had similar success with 100s of clients. Anecdotal evidence isn’t scientific, but substantial amounts of it should lead one to desire additional information.
To conclude, don’t assume the reported conclusions of a study are correctly reported and do not accept a study’s validity without digging a bit deeper. Being able to separate the wheat from the chaff when reviewing scientific evidence can help avoid being deceived.