How do you know if a health study is valid?

Almost every day you hear about another health study. This will cause cancer. That will make you obese. One thing is apparent:  you can’t rely on the media to check if the research is valid.  So, without being a statistician, how can you tell if a study has any merit? Today’s blog gives you some tips. 

• What does Google say? The recent study linking diet cola to strokes was all over the media. A Google search indicated the study most likely isn’t valid. Scanning the results on the first page, six  of eleven entries had the word ‘flawed’ or ‘controversial’ in the title or summary. But many times you won’t be that lucky. So here are some other tips.
• Self report or controlled study?  Self report is where study participants answer questions about what happended or how they feel (e.g., how many diet softdrinks do you drink every day? or on a scale of one to ten, how much pain do you feel?) The previously mentioned study was a self-report. Self reports are not always reliable.
• Do other studies corroborate? Are there other studies? Did most of them come up with the same result? If not, the study with different results may not be valid. For years people believed the study of Dr. Wakefield linking the Measles, Mumps and Rubella vaccine to autism. Dr. Wakefield’s study was the only study that found that link. Yet despite the fact that other research consistently came up with different results, the study was only recently retracted. ¹
• Planned or Retro?  Studies that were planned before collection of the data are more likely to have valid results than studies where information is collected after the fact (retro). In a planned study, the  researcher can try to control to limit or eliminate other factors which may also contribute to the results being studied.²
• Is the sample good? The sample is a subset of the entire population being studied. You may not have access to information about how the sample was selected, but if you do, here’s a couple of things to look for. First, a random sample increases the likelihood that the results will be valid. Second, is the sample representative of the population the study is making inferences about? For example, if the sample included only diabetics, it might not be valid to make inferences about the general population. One study showed that eating 1 2/3 tablespoons of safflower oil a day helped people lose weight and reduce waist size. The statistical results were impressive. The study had the headline  “Can Your Salad Dressing help you keep the pounds off. New research suggests yes.” Everyone in the sample was diabetic.  The headline implies the results would hold true for the general population. That assumption may not be valid. It may be that diabetics process fat differently than the general population.³
•  What does your doctor say? Many of the studies you hear in the media are in the journals your doctor reads frequently. Ask your doctor his or her opinion of the study and if the results may apply to you.

References

1. How does a parent know if a published study’s findings are reliable and valid? Examiner.com Chicago
2. What makes a research study valid – Oley foundation
3. The Whole Art of Deduction – Research Skills for New Scientists,  Roger Marion, Ph.D.