WAS: What Adults are Saying about Polls--Part Two
News reports are unreliable. What to do?
In Part One we revealed public opinion polling was a common staple used by media to frame and prioritize reporting of the days events. This is especially so in a political campaign season.
Readers also learned that too few news folks dispensing poll results exercise enough care in leveraging poll data. While limits to poll data typically are at least partially disclosed (all polls have these) they are often forgotten in story dialogue. This mean far too often conclusions are portrayed more definitively than can be supported by available data.
In this, part two, readers will discover media dependency on polling data is astonishingly high, and often, used as a crutch in lieu of more labor or brain-intensive news gathering and production practices. This creates a blizzard of so-called “poll news”, which by the time it reaches the consumer has been so heavily filtered and massaged, often fuels more uncertainty than clarity.
This is partially driven by the catchall phrase “margin of error” often used as a “throwaway” line to acknowledge all polls have interpretative limits. But such error is multi-dimensional and goes beyond “ plus or minus 3 points.” An equally important concept in the level of confidence—which is almost never reported. This factor expresses how certain a pollster is of results. In other words think of a weather forecaster proclaiming “today it will rain.” To you,—the runner, farmer or operator of a high-end car wash—such info needs more precision. So a degree of certainty is expressed as in “there’s a 80% chance of rain today.”
Any poll is a sample of something to estimate the characteristics of a broader population. Unlike a census (something the US government must do once a decade as established by the Constitution), a sample does not attempt to count or engage everyone. In “estimating” a population (just like in forecasting the weather) establishing one’s certainty of a conclusion is critical. This is rarely found in any news using poll data and fuels an important comparison;
“Candidate white hat is ahead of candidate black hat by 49 to 42 %. This poll has a margin of error of 3%.”
A proper reporting of these results should be:
“Candidate white hat is ahead of candidate black hat by 49 to 42 %. This poll has a margin of error of 3% at the 95% confidence level”.
The reasons why this is not reported are not fully understood. No student finishing a graduate degree or market researcher presenting to your company’s CEO would omit this. But in news presentations it’s almost never found—not even in the “fine print”—across just about every medium and platform. This omission is critical for at least the following reasons.
Any confidence level also reveals the chance of inaccuracy. Just like a home-plate umpire doesn’t judge every pitch correctly, so too are samples. At the 95% level (a common but not universal standard) a one-in-twenty chance exists the sample is wrong and conclusions drawn from it incorrect; at the 90% level one-in-ten; at the 80% level one-in five. You get the idea.
As a confidence level is lowered by a pollster for any number of reasons (cost, teasing out differences, etc.) so is the numeric margin of error. This means results which “overlap” can be made distinctive by altering confidence level. But doing this raises the chance that the entire sample is wrong. Any legitimate pollster learns early this mathematical sleight of hand and is entrusted to not abuse it. But if nothing else modern newsrooms are voracious content machines requiring constant feeding. A network did not spend good money on a poll to learn there are no conclusions and thus a series of endless trade-offs starts between the science of polling and the demands of news commerce.
You might be thinking pollsters are no different than palm readers! Not true. The science of polling is fine and endlessly tested and retested to prove it. It’s the interpretation and application of the data generated that widely befuddles efforts to gain clarity. Newsrooms vend news and in doing must avoid being sure aren’t data frauds, but as long as what it reported is plausible, the pollster’s highest concern that findings are probable, merit little concern. If you are still reading congrats! You now deserve to know what to do amid this melange of data, equivocation, punditry speculation, campaign spin and all the other things that drive us nuts in the first place when everybody on the news starts yapping about polls.
So consider this.
These journalistic data blizzards have invited a certain type of pollster who simply takes all publicly available info and makes a sort of “average” as way to enhance precision. This approach is mathematically justified and identical to the reason why casinos can never lose money in the long run (unless you are a certain Presidential candidate). It’s based on the proven idea the more something is measured the more normally distributed it becomes (the old bell curve again). A sharp stats person with good polling chops can effectively leverage this science and provide greater clarity even in a very tight election (something that is true as of this writing). Two that I most recommend are the folks at the Princeton Election Consortium and perhaps the most important political pollster to emerge over the 20 years in Nate Silver.
The folks at Princeton are foremost professors and data scientists not pollsters or news hacks. While this means they don’t avail themselves to the 24-hour news cycle everyday it does mean when they do update their models, they have something noteworthy to report. What’s better is they simultaneously focus on not only the White House, but the Senate and the House too.
Writers here are also committed to providing a running tabulation of sorts that shows today’s most likely outcomes at a macro level. For the record at the time of this writing, a slim victory is predicted for Harris, a notable, but not landslide, majority for the Democrats in the House and a slim, one seat GOP majority in the Senate.
To the degree a self-avowed “nerd” can be a polling rock-star, Silver is it! The child of academics, his pedigree lends itself well to his profession. In his “modeling” approaches he most seeks to establish the probability that a candidate will win (Like everyone he was “wrong” with his 2016 prediction however he did give Trump about a 30% chance to win on election day).
Perhaps most importantly in collating data he does something almost never seen in newsrooms by considering things like poll quality and reputation for impartiality and thus, does not weigh all polls equally. His proprietary approach is part of what makes him both popular and an industry opinion leader.
In this election, however, he seems intent on trying to raise his “business brand.” With frequency he has begun churning out analysis that reads far more like political punditry than stories that rightly gave him a favorable reputation. Plus he may be jumping the shark in creating content with far too much speculation and too little analysis. If any of this is happening this is a departure. Tread cautiously with him for now. But right after the September 10th debate maybe we can find him expressing his results with greater authority.
Hopefully, between these two sources you can gain the level of polling understanding you want while tuning out much of the superficial and repetitive coverage that is polling news.