Evidence: “It Worked for My Aunt Tillie” Is Not Enough

Evidence means different things to different people. Even quacks and their victims claim to have evidence that their treatments work. Sometimes that evidence consists only of testimonials from satisfied customers or from personal experience. “I tried X and I got better.” “I know Y works because it cured my Aunt Tillie’s arthritis.”

I had a friend who used all kinds of questionable treatments including homeopathy. I asked her how she decided what to try. She said if a friend told her something had worked for him, and if it didn’t seem dangerous, she would try it. That was all the evidence she needed. She didn’t care about scientific evidence because she said, “Science doesn’t know everything.” Comedian Dara Ó Briain had the perfect answer to that: “Science knows it doesn’t know everything; otherwise, it’d stop. But just because science doesn’t know everything doesn’t mean you can fill in the gaps with whatever fairy tale most appeals to you.” When Oprah Winfrey told Jenny McCarthy that experts said there was no scientific evidence that vaccines caused autism, Jenny retorted, “My science is named Evan, and he’s at home. That’s my science.”

There is a huge disconnect between what science-based medicine calls evidence and what alternative medicine and the general public call evidence. They are using the same word, but speaking a different language, making communication next to impossible.

First, there is no such thing as “alternative medicine.” There is only medicine that has been tested and proven to work and medicine that hasn’t. If a treatment currently considered to be alternative were adequately tested and proven to work, it would be incorporated into mainstream medical practice and could no longer be considered “alternative.” It would become just “medicine.” So-called “alternative” medicine can be defined as medicine that isn’t supported by good enough evidence to earn it a place in mainstream medicine.

“Alternative medicine,” along with “complementary and alternative medicine” (CAM) and “integrative medicine,” is not a meaningful scientific term, but a marketing term created to lend respectability to things that we used to call by less respectable names like quackery, folk medicine, and fringe medicine. It puts everything outside of science-based medicine under a single umbrella, including things that might work but haven’t been adequately tested, things that have been tested and proven not to work but that are still being used (like applied kinesiology—a bogus muscle testing procedure used by many chiropractors), and fanciful things that couldn’t possibly work, like homeopathy. It fosters the misleading impression that all these things might be equally valid.

Science-based medicine has one rigorous standard of evidence, the kind of evidence government agencies require before they allow a pharmaceutical to be sold. CAM has a double standard. They gladly accept a lower standard of evidence for treatments they believe in. However, I suspect they would reject a pharmaceutical if it were approved for marketing on the kind of evidence they accept for CAM.

When science-based medicine evaluates the evidence for alternative medicine, here is what it concludes:

• Acupuncture is just a theatrical placebo.
• Homeopathy not only doesn’t work but couldn’t possibly work.
• Chiropractic is essentially physical therapy contaminated with bogus diagnostic and treatment methods.
• Energy medicine is fantasy medicine dressed up with sciencey-sounding words like “quantum” and “frequencies.”
• Naturopathy claims to stress prevention and to consider the whole patient, but that’s just what every good doctor does. It offers “natural” treatments that have not been tested or that have been proven not to work, like homeopathy. What naturopaths do that is good is not special and what they do that is special is not good.
• Herbal medicine is plausible: after all, half of our prescription drugs came from plants. But every herbal remedy must be tested individually to determine if it is effective and safe; and statistically the great majority of promising remedies fail testing.

So while science says the evidence is lacking, CAM says there is plenty of evidence:

• They cite preclinical studies in animals and test tubes, but scientists recognize that even the most promising preclinical studies can be misleading and must be confirmed in humans.
• They cite case reports, but we know that case reports are only useful as indicators of what science should evaluate with controlled studies.
• Sometimes they can cite clinical studies in humans, but those studies are often flawed. They may be poorly designed or lack a control group, they may be contradicted by other studies, they may be based on such extraordinary claims that we would have to have extraordinary evidence to accept them, and they don’t build on each other to create a coherent body of evidence.
• They cite doctors who say, “In my experience, X is effective.” Mark Crislip, one of my colleagues on the Science-Based Medicine blog, says “in my experience” are the three most dangerous words in medicine, because personal experience is so very compelling and so often leads to false beliefs.
• They rely heavily on testimonials and anecdotal evidence. Scientists know that the plural of anecdote is not data; no matter how many testimonials you accumulate, they can’t ever prove that the treatment works. Think of all the testimonials over the centuries for balancing the humors with bloodletting.

CAM believers find their evidence very convincing, and it is hard to explain why we don’t and how their kind of evidence has been shown over and over to lead people to false conclusions.

You can’t blame people for accepting that kind of evidence. They are only doing what comes naturally. Science doesn’t come naturally to humans. Evolution equipped us with thinking processes that facilitated survival on the African savannah. Our ancestors had only two sources of information: their own experience and the experiences of other people. If someone said “Don’t eat those berries; they are poisonous,” there was no way to verify the information; but listening to others was likely to help keep you alive. Humans became very adept at pattern recognition. If a pattern of shadows in the bushes looked sort of like a lion, it was safer to assume it was a lion and run away. The cost of over-interpreting patterns was small, while missing the pattern of a real lion could be fatal. Our ancestors learned to make quick decisions: if it was really a lion, they needed to run away now. And emotions served as motivators: fear made them run away faster.

Those thinking processes helped our ancestors survive in a prehistoric environment, but they have become a handicap in the modern world. Today we have more sources of information, but our minds still work the old way. We prefer stories to studies, anecdotes to analyses. We see patterns where none exist. We jump to false conclusions based on insufficient evidence. Emotions trump facts. If your neighbor had a bad experience with a Toyota, you’re likely to remember his story and not buy a Toyota even if Consumer Reports says it’s the most reliable brand. That isn’t logical, but humans are not Vulcans. When we act illogically, we’re just doing what evolution has equipped us to do. It takes a lot of education and discipline to overcome our natural tendencies, and not everyone can do it.

Ray Hyman is a psychologist and one of the founders of modern skepticism. When I asked him why some people become skeptics and others don’t, he said he thinks skeptics are mutants: something has evolved in our brains to facilitate critical thinking. Is it nature or nurture? Or both? Maybe life experiences also incline some of us to think more critically.

So what kind of evidence should persuade us? It’s tricky, because, as John Ioannidis has shown us, many published research findings are wrong. Early pilot studies are overturned by larger, better-designed studies. Studies are influenced by researcher bias. Some are even fraudulent. With over 23,000 scientific journals and 700,000 papers published every year, it’s easy to find a study to support almost any belief.

In this context, there is a hierarchy of evidence:

• Basic science.
• Test tube studies (in vitro).
• Animal studies (in vivo).
• Case reports—of a single patient.
• Case series—reporting on a number of patients.
• Case control studies (example: comparing people with and without lung cancer to see if there are more smokers in the group with lung cancer).
• Cohort studies (example: following people who smoke and who don’t smoke over a period of time to see which group develops more lung cancers).
• Epidemiologic studies (example: studying whether people in countries with more smokers develop more lung cancers. These studies can show correlations, but they can’t determine causation. Countries with more smokers might have other confounding factors that predispose to lung cancer).
• Randomized controlled trials (RCTs).
• Systematic reviews and meta-analyses that evaluate all the published evidence pro and con.

The best studies are large, randomized, placebo-controlled, and double- blinded; but that isn’t always necessary or even possible. We don’t need an RCT to know that parachutes save lives. It’s unethical to knowingly endanger a control group by denying them effective treatment. (Anyway, who would volunteer to jump without a parachute?) We didn’t need RCTs to figure out that smoking causes lung cancer because there is an overwhelming body of corroborating evidence from several different kinds of research.

Evidence matters. Science works. It’s a collaborative, self-correcting enterprise. It never relies on a single study. When a finding is true, it will be corroborated by other studies, and a consensus will eventually build based on a cohesive body of evidence. False findings are eventually overturned and discarded. We can trust the scientific method; we can’t trust anecdotes. Sorry, Aunt Tillie!

This article was originally published as a SkepDoc column in Skeptic magazine.