A Study In Studying A Study

You are probably aware, from your having had to sit through endless political shouting matches in the media, that a story can take on some wildly different tones depending on who is reporting it and how they view the event in question. What I provide today isn't political at all, but just because something isn't political doesn't mean it won't be similarly parsed.

The source story is an article published in the journal Cancer assessing the effectiveness of acupuncture. In case you need a refresher, in acupuncture, the object is to stimulate specific areas of the body in order to improve the flow of chi through it by puncturing those areas with needles. The full article's behind a paywall, but in summary, the study, led by Ting Bao of the University of Maryland, took 47 women who had breast cancer ranging from stages 0-3 who were taking aromatase inhibitors, which help treat the cancer but may also induce menopause-like side effects (women taking them are already post-menopausal). 23 of the women were given an acupuncture treatment, but the other 24 were given what was deemed a sham-acupuncture treatment, in which the needles were placed more or less randomly and not actually inserted into the body. The study found no significant difference in how the subjects responded to either treatment; both responded just as positively. The conclusion was that, while either method showed a positive response, acupuncture's effect was akin to a placebo.

Going through previous studies on the matter, this study is consistent with the majority of previous studies I located, though there are certainly studies that disagree and have found the real acupuncture to have outperformed the sham acupuncture. I can't really be sure which side constitutes the true majority. It appears that even the act of studying it requires its own studies into how the sham acupuncture should be carried out (for one, the person administering the acupuncture may be biased); neither Bao nor Patricia Ganz of UCLA, who are interviewed here, think this particular study to be conclusive and agree that more and larger studies need to be done, though Bao admits it's probably worth a shot if you're in that situation. 47 people isn't all that great a sample size, and in science, you're not done until you can consistently repeat results, something that isn't being done here. The abstract of the source article further declares a need for additional study specifically on differences in response levels among different ethnicities, as African-Americans responded comparatively better to the sham treatment.

That's about the most even-handed summary I'm able to manage.

Enjoy it. You won't see much more of it. Now it's time to go through an assortment of other writeups on the study.

*The HealthFinder article linked above, referencing Ganz, went with the headline "Acupuncture No Better Than 'Sham' Version in Breast-Cancer Drug Study" and waited until the main body of the article to start introducing ambiguity to the matter. Though once it does, the body of the piece heavily emphasizes the need for further study.
*An obscure place called PR Pick, meanwhile, characterizes it as "Both Real And 'Sham' Acupuncture Reduce Side Effects Of Breast Cancer Chemo", and wonders whether it matters whether it's real or not, though it does eventually get around, in the late stages of the piece, to allowing that the sample size was small- PR Pick specifies that there were only nine African-Americans in the study- and that, yes, further study is warranted. At least it made it in there.
*Lauren F. Friedman, for SFGate via Business Insider, gets a tad bit starker for the 'anti' side with the headline: "New Study Exposes Acupuncture As Pseudoscience". There's nothing in this article about requesting further study, even though it links the HealthFinder article.
*Nicholas Bakalar of the New York Times takes the 'pro' side with his interpretation: "Acupuncture, Real or Not, Eases Side Effects of Cancer Drugs". He emphasizes Bao's comment that it's worth a shot and, again, does not mention a need for further study.
*Jaleesa Baulkman of the University Herald goes so far as to praise the sham acupuncture in her headline, "'Sham' Acupuncture Treatments Can Ease Hot Flashes, Side Effects Of Breast Cancer Treatments", focusing on the specific areas where the real acupuncture and the sham acupuncture improved on not having acupuncture. And- once again- there is no mention of the need for further study.

Most of the articles written on the topic take the 'pro' side of the argument over acupuncture, emphasizing that they both produced positive results over and above the fact that neither did any better than the other. Which is fair enough; the results did show that, after all. It's not like much of anyone's coming out and saying acupuncture's going to kill you or anything; at worst it's pretty much harmless as long as you're using clean needles. Many, though, did not also feel the need to call for further study: the New York Times did not, while PR Pick of all places did. The places that called for further study earliest and most prominently were industry-related publications that have been through this rodeo before, such as News-Medical ("Acupuncture treatments may help alleviate side effects of breast cancer drugs"- note the word "may" in there) and PsychCentral, where the very first words in the article were "A new pilot study", aka 'this study is hopefully the first of many'. These two pieces are most interested in the possibility that the sham treatments themselves may be beneficial in their own right, without outright implying that they are, as the University Herald piece did. 

Think of acupuncture what you will. Personally I don't think much of it on the basis that I don't need perfectly good skin getting needles poked into it by the bushel, though I allow that placebos can have a surprising effect on people. Your brain can trick you into one hell of a lot, for better or worse. But whatever you make of it, the only truly responsible way to report on this study in particular is to emphasize, as early as possible, that the study didn't tell the whole story and the study leader said as much. Running full-speed in one direction or the other and acting like this is the end of the story is not what you do here. 

And when PR Pick understands that and the New York Times doesn't, we have some issues.