Explanatory studies are done under controlled conditions to determine whether a treatment has any efficacy compared to a placebo. Pragmatic studies are designed to assess how the intervention performs in everyday real world practice. Pragmatic studies measure practical success but don’t determine actual efficacy: that requires a proper randomized controlled trial (RCT) with an appropriate control. Pragmatic studies have their place, but they can sometimes make an ineffective treatment look good: a phenomenon I have christened Cinderella Medicine.
Research studies don’t always predict how a treatment will perform post-marketing. A study might be done on men aged 30 to 70 with no other illnesses. Once the drug is out on the market, it will probably be taken by women, by people who are under 30 or over 70, and by those who have concurrent diseases like diabetes or atherosclerosis or are taking a lot of other medications; and the dose or frequency may vary from the study protocol.
A good example is the clot-busting drug t-PA. In clinical trials, it clearly improved the outcome of stroke patients, but in actual use in community ERs the death rate was almost twice as high in patients who received it as in patients who did not. There are many possible explanations: perhaps it was offered to patients with more severe strokes, perhaps the protocol wasn’t followed as carefully, perhaps some patients were mis-diagnosed, perhaps they had confounding factors that were not present in the research studies.
Pragmatic studies are useful for the questions they are designed to answer: how a treatment performs outside the limited environment of a research study, which of two treatments should be preferred by doctors, patients, and policymakers, deciding how limited resources can be best utilized. They are an integral part of comparative effectiveness research.
But they have limitations. They are unblinded and the patients are generally self-selected. Larger studies are needed to compare two active treatments than to compare an active treatment to a placebo. Lack of compliance and a high drop-out rate can skew results. Patient self-assessments of subjective outcomes are not as reliable as objectively measurable outcomes.
Pragmatic trials can’t determine which components of a “package” of care are essential, and they can’t assess the contributions of the therapeutic relationship. When used for studying a treatment with a strong placebo effect, they may make an ineffective treatment look better than an effective one. CAM proponents like pragmatic studies because they are often the only studies that seem to support them. They are attempting to bypass good science by showing that, in practice, their methods seem to work.
Cinderella didn’t look very pretty in her rags and ashes. Imagine that her Ugly Stepsister had a complete makeover, with hair styling, expertly applied cosmetics, jewels, and a beautiful designer dress. Maybe tooth whitening or even orthodontia, charm school, modeling classes, and elocution lessons. If you entered her in a beauty contest along with the unadorned, dirty Cinderella in her original rags, the Ugly Stepsister might win hands down. But it wouldn’t be a fair beauty contest unless both were in their original unenhanced state or unless you compared the makeover-enhanced stepsister to the Fairy-Godmother-enhanced Cinderella. (Or if, for a really well-controlled study, you managed to persuade the FG to do her magic on both of them.)
Acupuncture’s makeover
Studies comparing acupuncture to standard treatment have shown that acupuncture works better. Standard treatment is like the original Cinderella in her rags: plain and unenhanced in any way. The doctor may only see the patient for one visit and tell him “You have a common garden variety backache; we don’t know why people get these, but they usually resolve spontaneously in a few weeks; while it is going away on its own, I could offer you a prescription for pain pills or a referral to physical therapy.” He doesn’t spend much time with the patient and may seem bored and unsympathetic.
Acupuncture is like the Ugly Stepsister after her complete makeover. The treatment itself (insertion of needles) is like the Ugly Stepsister before her makeover. It doesn’t have any specific effects (it is no more effective than touching the skin with a toothpick). But the acupuncturist surrounds the treatment with all kinds of enhancements that produce “nonspecific effects” that are not due to the treatment itself, but rather to the interaction with the provider.
- The acupuncturist assures the patient that he knows how to relieve the back pain, and he provides a complicated explanation with all kinds of impressive, esoteric oriental terminology.
- He explains that his system is derived from ancient Chinese wisdom and that his needles will adjust the flow of qi through the patient’s meridians to restore health.
- He takes the patient into a quiet back room, has him lie down and relax, and spends half an hour or more doing up-close-and-personal hands-on treatment.
- He is charismatic, shows great interest in the patient, asks a lot of questions, and may uncover another unrelated problem that needs treatment.
- After treatment, he prompts “You feel better now, don’t you?” and the patient feels a social pressure to agree.
- Instead of dismissing the patient with a prescription, he asks him to return over and over, maybe 3 times a week for several weeks; and when the initial course of treatment is finished, he may want the patient to keep seeing him for treatments to maintain health and prevent future problems. He develops a strong, caring doctor-patient relationship.
The plain needle insertion has been given the Cinderella treatment and transformed into an enhanced package of suggestion, expectation, relaxation, ancillary psychological effects, personal interactions, etc. Acupuncture treatment is ready to go to the ball and wow the prince. It is the ideal placebo package; it’s hard to imagine how anything could be devised that would be better at eliciting placebo responses.
So acupuncture, with no specific effects but many nonspecific treatment effects, will appear to outperform a standard treatment that offers some small specific effects but little in the way of nonspecific enhancements.
The common argument is that it doesn’t matter how a treatment succeeds: it should be used because patients report feeling better faster than with standard care, that it is more effective in a practical sense. But fooling patients with nonsense about imaginary meridians, qi, and acupuncture points amounts to lying. Using placebos and offering fantastic explanations undermine the doctor-patient relationship; and this kind of thing leads people to think non-critically, to accept other kinds of pseudoscience, and to reject science-based treatments that might help them objectively or even save their life in the future.
Cinderella treatments: beautified and dressed up with added enhancements. Why not add as many as possible of these Cinderella enhancements to standard, science-based treatments? Without lying or misrepresenting our knowledge? These pragmatic trials don’t show that acupuncture works; they show that the way standard treatments are offered provides fewer nonspecific effects and could stand a bit of a makeover. That should be our goal.
This article was originally published in the Science-Based Medicine Blog
