Are polls an accurate view of American public opinion? This morning I ran across a story touting an ABC/Ipsos poll. Titled “Most Americans blame Vladimir Putin, oil companies for high gas prices: POLL”. The opening paragraph states: As politicians spar over who’s to blame for recent increases in gas prices, a large majority of Americans say oil companies and Russian President Vladimir Putin are major culprits, a new ABC News/Ipsos poll finds. This statement closely follows the current White House and Democratic party lines that oil companies are once again greedy and it’s all Putin’s fault anyway.
Of course, the same poll by Ipsos is cited by ABC, CNN, Yahoo News, and Politico, along with others. It’s hard to see how rising gas prices can be Putin’s fault given that prices started to rise in January 2021 and rose steadily throughout last year. They spiked in March after Putin invaded Ukraine, but prices had been rising long before that. So, no. It’s not Putin’s fault.
As I usually do, I went immediately to the end of the story to see how many people they polled, the political breakdown of those respondents (how many of each party), and most importantly, the margin of error. Unsurprisingly I found problems that would get any article I submitted to an academic journal rejected. And I’d likely get yelled at for my sloppy survey techniques.
First of all, they’re basing their analysis on results from 530 people and extrapolating that out to roughly 257 million (adults of voting age). Those who analyze polls and teach statistics (like me) will tell you that to be even somewhat accurate in polling a country as large as the U.S., one should aim for at least 1,500 respondents. Now, I recognize that there are people who are going to scream that that number is far too small as well. But it does work statistically and a hell of a lot better than 530. Bear in mind, all polls are approximations and are not designed to predict what any individual will say or do. They are a methodology that will give a general feel of the leanings of groups of people. So, I can safely say that to be in any way mildly accurate in their assessment of American leanings, this poll should have had about three times the number of respondents.
Next, let’s look at the margin of error. According to Ipsos it’s 4.9% and within the 95% confidence interval. Roughly speaking, that means you can add 4.9 to the top percentage and subtract from the lower end to get the full range of possible support for a question. For example: this poll suggests that 71% of Americans blame Putin for high gas prices. So, Ipsos is 95% confident that the actual number of people who blame Putin is between 66.1% and 75.9%. That means between two-thirds to three-quarters of Americans think it’s Putin’s fault. That’s a pretty big swing. And for that reason, when using stats in social science we tend to avoid margins of error greater than 4% at the outside and even then any journal article is filled with caveats.
The poll did get the party division right, at least according to Gallup which shows that roughly 42% self-identify as independent, with 26% identifying as Republican and 29% as Democrat. However, Gallup also asks independents if they are Republican-leaning or Democrat-leaning. With those numbers we get 9% non-leaning independent, 42% Democrat-leaning independent, and 47% Republican-leaning independent. Depending on how questions are asked, this can become a deciding factor in how individuals reply to questions. This also skews the Ipsos analysis of how Democrats and Republicans responded versus independents.
Ipsos also didn’t provide any information on the ages or voting status (e.g. are they registered to vote, did they vote in the last election) of their respondents. Since they’re carefully telling us about the partisan leanings of respondents, they should also have no problem telling us the age ranges and voter status.
Finally, when I went to the Ipsos site and their reporting on this survey, I discovered that it was an internet-based poll in which Ipsos selected the respondents. In other words, not a truly random sampling. Yes, invitations to participate were random, but then they say, “those selected who do not already have internet access are provided a tablet and internet connection at no cost to the panel member. Those who join the panel and who are selected to participate in a survey are sent a unique password-protected log-in used to complete surveys online.” It’s not really clear how they selected individuals to participate, and I have to wonder about that. Again, if I did a survey this way, I would have to fully explain how I made sure that those who responded were truly a random sample.
Additionally, there is nothing about how many people were originally contacted, so we have no idea what response rate is represented by that 530. Is that 100%? 25%? Again, general rules about survey results state that a response rate of less than 35% is highly problematic and not going to be published by a top-tier journal. Not telling us how many households were originally contacted and the response rate suggests that Ipsos simply ran with the responses they got and didn’t bother to try to get more.
In the end, I’m very skeptical of these poll results (as I am with almost every poll I see). On top of all the statistical issues I discussed above, you also have to remember that people flat-out lie on polls for a variety of reasons including wanting to deliberately skew numbers, not knowing but answering anyway, and not wanting to “look bad” to the survey taker (and that’s true even for on-line surveys). Given today’s climate of doxing and cancel culture, I also wouldn’t be surprised if people are hesitant to respond honestly to an online poll for fear that results will be hacked and tracked back to them thus putting them in jeopardy of losing their job, or worse.
So how do you get an idea of the general feelings of the American public? Well, you can spend your own time averaging results. You can read a variety of sources and compare analyses. Or you can just go the opposite of whatever the major media outlets tell you (including the ones you might feel are closer to your preferences). I can suggest that Real Clear Politics does a decent job of averaging polls on broader questions.
For any particular poll or article about a poll, pay close attention to the words used to describe and discuss the results. I used to tell my students that social science academics are very good at using what I called “weasel words.” Words like “appears” “seems” “suggests”. Or hyping how many people agreed with the Democratic party line if something is viewed as a good policy, and how many Republicans agreed with a policy that was viewed as “bad.”
In the end, polls are a very general snapshot of public opinion. They are influenced by the day of the week, time of day, international and domestic events, and even good and bad days experienced by respondents when they replied. Just like you can have a snapshot of what looks like a fantastic day at the beach with your friends, but right after that you all had a huge falling out and nobody speaks to anybody anymore. The snapshot is not an accurate representation of your relationship even the next day. It may be, but by only looking at the photo, we have no way of knowing. Same thing with polls.
So, do Americans blame Putin and oil companies for high prices? Not likely. For one, Americans aren’t as stupid as Democratic Party operatives would like to believe. People do know how much they’ve been spending on gas and when that started to increase. Secondly, the White House has once again turned to Tik-Tok “influencers” to get the word out that it’s all Putin’s fault. That alone should make you question the veracity of the claim. Finally, gas prices started to rise in January 2021 and Putin didn’t invade Ukraine until February 24, 2022. Yeah, Putin is responsible for a lot, but not this one. This one is all on Biden.
If you get challenged with the “but oil companies are screwing us over! Reeee!” ask your challenger why the oil companies “let” prices get so low prior to 2022? Were they just being nice and then decided they would be evil again? Are they evil capitalists on an every-other-year basis? How do they decide when to be evil and when to be nice? These are questions that will make unthinking heads explode.
Take all polls with a large shaker full of salt. But maintain a HUGE amount of skepticism about polls that come out just in time to support the current theme emerging from this White House administration.
Trust your eyes and your gut. They’re not lying.
An issue with Gallup– they’ve been having “issues” since at least the mid-teens, and while I’m too tired to find it again, I seem to remember they shifted strongly to a sampling that was largely online, self-selected, for pay, and had massively lower turn-over since then.
I’m delighted you went through all the ways the poll is wrong, even though I would’ve stopped* when I looked at the stated question about how much you think they’re to blame, because I *know* that Putin has a lot of blame for the oil nonsense because of funding stupid propaganda for decades, now, and that’s just the known stuff.
(the anti-fracking nonsense was largely Russian funded)
* ….OK, probably would’ve stopped, depending on how it hit my Elephant Child tendencies. >.>
It’s one of my bugaboos since I taught stats for 20+ years…
:chuckles and cheers:
It is well worth petting that peeve.
Heh. I self-selected for Gallup. I was never offered any sort of recompense, but I *was* aggressively purged from their system after answering only three polls*.
*As expected, the polls were commissioned to find a preselected answer, and the questions were heavily slanted to advance that cause. My ejection strongly implies that sample selection was also intentionally on tilt.
Very good.
Thanks! I was trying not to get too ranty…
Rant all you like, it’s your blog. Besides, I like a good rant. I follow The Rageaholic after all
Concur with the others, Professor. Very thorough and completely on target.
What does the size of the sample matter when you’re only looking to confirm the approved propaganda?
im surprised they don’t just conduct anonymous surveys at MSNBC studios or DNC headquarters and declare it fair because it was an anonymous sampling.
I want to see the questions. Certainly I blame Putin for high gas prices. About 20% belongs to him. 70% belongs to Biden’s dumbass policy decisions, and 10% belongs to the oil companies for not realizing quickly enough that oil was on the way up and going wide open in the areas they could, because oil was going high enough to make marginal wells profitable.
During the Hillary campaign in Ohio we were inundated with supposed telephone polling calls. They would call again and again; 2 and 3 times in the same day. I finally resorted to lying my gluten maximus off by responding that I was a yellow dog Democrat who just loved Hillary and everything on her platform . The calls stopped and the Hillary campaign left the state. I later discovered my co-workers were doing the same thing.