Wednesday, July 23, 2014

Adventures In Polling

Yesterday, Walt Hickey of 538 ran a fun little thing about the popularity of various Star Wars movies and characters. They ran the survey themselves online via a place called SurveyMonkey, which should probably tell you something about the scientific validity, but it's just Star Wars and not a Presidential horserace poll or anything really important so who cares, honestly. Among the things we 'learned': people like the original trilogy more than the new one, Luke Skywalker is the most popular character and, as you'd probably expect, Jar Jar Binks is the least popular. (Also learned: 39% of respondents thought Han shot first, with 24% thinking Greedo shot first. Han did in the original; Greedo had a first shot inserted in the 1997 remake.)

Fun. Silly.

And then Christopher Ingraham of the Washington Post got hold of it. As far as I can tell, he looks to be the genesis here, though it's possible that someone else got there first. Ingraham decided to compare 538's popularity ratings of the Star Wars characters with popularity ratings of actual, flesh-and-blood politicians, and of course nobody in real life did too well. He opted to chop out the ratings of Princess Leia, Obi-Wan Kenobi, R2-D2, C-3P0, Anakin Skywalker, Lando Calrissian and Padme Amidala- all of which were more popular than all the humans- and went straight for the high-rating Luke, Han Solo and Yoda, mid-table Darth Vader (of course he's midtable, because Vader's a badass), and the lowest-ranking character that beat or tied all the humans, Boba Fett (who drew even with top human Hillary Clinton). And then the humans get rattled off, with Emperor Palpatine wedged in between Marco Rubio and Paul Ryan, more humans, and then Jar-Jar slotting in between Rick Santorum and John Boehner, and then the rest of the human contingent, with 'Congress' bringing up the rear.

And then Ingraham, clearly flying into the 'if 538 said it you can accept it at face value' bug zapper, goes right for the headline "Darth Vader is polling higher than all potential 2016 presidential candidates". Ha ha, how wacky, weep for the smoldering ruins of your country, America, be afraid, be very afraid. And of course the fringe blog set went and had their fun, though USA Today piled on too, because 538 yo, they're NEVER wrong. I'm not even going to reward the rest of them with a link.

Because this entire premise is founded on crap. You'll notice I haven't given the actual numbers of the Star Wars characters and real-life politicians. That is why. They mean nothing sitting next to each other and it would be adding misinformation to the world to put them next to each other. So I won't do it.

The problem is simple. If you are going to compare approval ratings of disparate entities such as this, you need to do it in the same poll, asking the same group of people. That is the only way to eliminate all the variables, up to and including what the respondents think when they're asked about the relative popularity of real and fictional people. (Never mind that the respondents could very well start giving joke answers once they see what the survey is trying to pull on them, because respondents often view such enterprises as wastes of their time and answer accordingly.)

538 was using a fairly informal poll on SurveyMonkey (even if they did supply the raw data). The actual politicians were polled seriously, using Gallup and using what is almost certainly a 100% different group of people, of wildly different demographics. The two simply do not equate. You've basically taken some random one-off Internet poll and mistaken it for the Pulse Of The Nation.

This just frustrates me to no end. There are people that bust their asses, day in and day out, to make numbers mean something. People that crunch data, make double-super-positive sure that the mix of people they're asking is reflective of their target constituency, explain it in the most accessible and thorough manner possible. Many of those people work for 538, which is why even a silly poll such as this came with the raw data of every single respondent attached, and why they were asked basic demographic questions: home region, income level, education level, age and gender. For all that data to be completely flushed down the toilet because some guy ignored all the numbers except Darth Vader's end approval rating in one poll, the presidential candidates' approval ratings in a completely different poll, and the number 538, and posted it and had people take those disparate end numbers at face value drives me crazy.

56% of those responding to a phone survey in Ellensburg, Washington favored the building of a new middle school. Meanwhile, 56% of respondents in a poll of MLB players before this season declared Mike Trout as the best player in baseball. Therefore, Mike Trout and a middle school in Ellensburg, Washington are equally popular, right?

The difference between my comparison and Ingraham's is that you can more plainly see how stupid mine is.

No comments: