“Lies, damn lies, and statistics”. Perhaps it is time to update the quote attributed to Disraeli. I’d like to suggest, “Lies, pure BS, and polls”..Every few weeks some new poll reports what Canadians or Albertans are thinking. Please don’t be fooled... the intention of this reporting is not to inform you what others are thinking. No, it is intended to change your opinion without you having to do any serious thinking for yourself. So, perhaps if you can spare a few minutes to read further, you might examine the next published polling results with a more critical mind..The methods of applied statistics are well understood and of huge benefits where quantitative measurement is impossible and so sampling a population is required. Take for example the classic problem of estimating the number of black marbles in a jar of white and black marbles. You could empty the jar and count every marble. Or you could take a few dozen out of the jar, count the number of whites and blacks and make a conclusion on the entire population. Statistics will tell you how confident you can be about your conclusions — this is the “+/- 2.8%, 19 times out of 20.” All of this is fine. But where it gets a bit murky is when humans are surveyed about their opinions, and then we try to predict election outcomes, or try to influence government policy formation based on these opinions. This is where you need to be s bit more sceptical of what you are reading..Is counting opinions that much different than counting marbles? What could go wrong? Let me count the ways..The difficulty for pollsters is to frame neutral questions that don’t in the very asking, prejudice the answers. Modern psychological research shows how difficult this can be. Author Malcolm Gladwell in his book 'Blink' explores both the benefits and the dark side of our instantaneous reactions. He explains how not only our attitudes and opinions can be ‘primed’ by words and images, but also our performance. It seems that if you are asked to think about words like ‘professor’, ‘intelligence’, and ‘wisdom’ before you play Trivial Pursuit, you will actually score higher than if you are asked to think about words like ‘soccer hooligans’, ‘stupid’ and ‘fool-hardy’. Psychologists conclude that our unconscious attitudes revealed in a ‘blink’ may be completely incompatible with our stated conscious values. Further, the opinions we profess today may well be due to the priming we heard in the last week or the last year which may have never been filtered by actual critical thinking..If you think the psychologists are of little help, then don’t look to the economists. Like the author of the biblical Book of James, economists think it is not what we believe or say that is important, but rather what we do. The only true preferences are revealed preferences; those preferences revealed by action. As James says, “faith by itself, if it is not accompanied by action, is dead”. My own personal preference in a car is a Jaguar and yet I drive a Toyota. My revealed preference when I actually have to make a real decision with real money in the real world is my actual real preference. This is why the question “If an election were held today ....” doesn’t really get at what I would actually do if there was a real election today. Thus the results may be suspect. But there are other issues as well..First, beware of partisan polls. These are polls that are undertaken to support a particular position. They are generally sponsored by a lobby group that is trying to change your opinion and thus claim public support for their preferred policy..For example, the RCMP union did a poll of Albertans in August 2022 about a move to a provincial police force. Not surprisingly, the pollster, Pollara, found that “Majority of Albertans opposed to provincial police force” to quote the Calgary Herald headline. The survey did acknowledge that more people where the RCMP were actually the police, favoured a move to a provincial police force. “What?” you say? Why were people who have their own police forces e.g. Calgary and Edmonton, asked about the police service that their rural neighbours should use? Why is it any of their business? It is a good question and points to a scientific error of the poll — they surveyed the wrong underlying population. It would have made as much sense to ask the residents of Montana the same question. Further, the Pollara poll contradicts an Angus Reid poll done at the same time that found that overall, people had less confidence in the RCMP than in their local provincial or municipal police force. Both these polls can’t be right. If a poll is partisan it should be suspect..Another example of a recent partisan poll is one done for Clean Energy Canada by Abacus Data. The lobby group wants to influence your thinking towards more “green and clean” energy options. Can you spot any priming going on here? But also, they are asking for people’s beliefs about matters of fact. Despite Albertans electricity costs rising due to the NDP forced phase out of coal fired electrical plants, half of them still think (or at least have been primed to think) that “clean” energy is more affordable. Also despite motor gas bills going up because of insecure global oil supplies, and our electricity grid nearly failing a number of times in sunless, windless December, half of Albertans still think clean is more secure. As the saying goes, there is no solution for insanity — 100% of surveyed people in court felt the emperor's new clothes were gorgeous..Second, beware of polls that survey the wrong underlying population (similar to the (intentional) RCMP union poll above.) This error was most apparent during the recent CPC and UCP leadership contests..The most exceptional thing about Pierre Poilievre’s win is that he obtained nearly 71% (68% of the points required) of the votes. This outstripped his polling numbers by 27-37% points as he was polling in the 34-44% range in the last six weeks before the vote. This is a polling error of 61-108% which is a far cry from “+/- 2.8%,19 times out of 20.” The reason for the error is all Canadians were polled, not just CPC members. But that didn’t prevent the numbers from being made public as if they were meaningful measures of who would become the next leader..The same thing happened to Danielle Smith. The Calgary Herald and Edmonton Journal hired Leger in July 2022 and September 2022 to poll Albertans concerning their voting preferences and their preferences for the leadership of the UCP. In July, Smith’s numbers by self-declared “UCP supporters”, but not necessarily UCP members, were listed as 22% followed by Brian Jean at 20% and Travis Toews at 15%. In September, in an attempt to simulate the ranked ballot voting, the first ballot numbers (again by those self identifying as UCP supporters) were Jean 31%, Toews 29%, and Smith 27%. Despite Don Braid knowing that actual UCP members were not surveyed he fearlessly lead his column “Brian Jean, premier of Alberta?” and said Smith “scores no higher than 31 per cent in any round (on Ballot 5) after starting out at just 27 per cent on the first ballot.”.The actual results on the first ballot were Smith 41.3%, Toews 29.4% and Jean 11.0 % - a 180% error. Again, beware if the survey population isn’t the actual people making the decisions..Third, remember that polls are not predictions. When the pollster asks “If an election were held today...” they are not trying to make a prediction on what will happen in the next election (except they let their readers make that leap). Further, if they were trying to make a prediction then they are asking the wrong question. Isn’t the proper question “When the next provincial election is held on May 29, 2023 what party ...”. Of course the right answer to this question is “Depends on what happens between now and then” But pollsters can’t get any headlines with results like “80% of active voters will wait for the election campaign to decide their votes” or “20% of voters will not be in town on election day” or “35% of Albertans are so disillusioned about our democratic institutions that they will never vote again”. (Remember, over 30% of eligible UCP members did not cast a ballot in what was arguably the most important leadership race since Jason Kenney bested Brian Jean.).If pollsters wanted more accuracy they would ask two important questions on any poll of voting intentions: 1. “Did you vote in the last election”? A ‘No’ answer would result in no further questions. 2. “Based on a scale from 1 to 10 where 1 is not determined at all and 10 is extremely determined... how determined are you to vote in the upcoming election”? Any number less than 6 would result in no further questions. Why would you survey people who do not participate in the democratic process about their political preferences or their policy preferences? Those preferences obviously won’t matter in the actual election. In a recent poll by Abacus Data, a self-titled “Alberta Politics Deep Dive” got detailed views on policy and “impressions” from people, 23% of whom “didn’t vote in the last election and likely won’t vote again”..This type of polling flaw shows up in all sorts of weird results. In the 2019 Alberta provincial election the UCP was consistently polling around 10% points above the NDP. However, when the votes were counted the UCP was 22% points ahead, a 120% polling error. Turns out the people answering the pollsters weren’t the people who actually took the time to vote..So what is person to do if they want to an idea about what might happen if an election were actually held today? Well, they have to dig a little deeper into the realm of polling aggregation and election modelling. In Canada, Eric Grenier provided this in the past but has moved on to be a CBC commentator. His role was taken over by Phillipe J. Fournier, who runs the website 338canada.com..The poll aggregators, as the name implies, put polling data together but weight it by its age and a quality ranking of the pollster. Then they integrate that data amongst other factors to predict election outcomes. The model methods are not secret and are explained on the website. The models work at the riding level but provide probabilistic outcomes for total popular vote, seat projection, the odds (probability) of winning the most seats and the odds of a majority or minority government. This is where you should look if you want to know what would likely happen, “if an election was held today.” Fournier has had pretty accurate results including the 2019 Alberta general election where he predicted 60 seats for the UCP and they won 63. Remember, he used the pollsters data which had more than 100% polling error for that election..So what does Fournier say now about the upcoming election? “Between now and election day, there are lots of variables at play.“ No kidding. But when he runs his models, the “day of” predictions consistently favour a UCP majority. When the model was run on December 9 the odds of a UCP majority was 66% which fell to 60% on December 23 and rose to 68% on January 29. The 6% drop on December 23 was due to the “Deep Dive” Abacus poll which concluded, “This snapshot of Alberta public opinion points to an advantage for the Alberta NDP and Rachel Notley in the early days of Danielle Smith’s time in office. The Alberta NDP has a lead in vote intentions, the more popular leader, and issues ownership on more of the top issues for voters.”.Sounds a bit partisan to me — are you convinced?.Part of Smith’s strategy is to try and win over voters who may be hesitant in accepting her re-birth in politics. She has somewhat less than four more months to work her magic..In the meantime, ignore the polls but occasionally look to see what Fournier is saying..William D. Marriot is a retired economist who studied way too much statistics and econometrics while at university.
“Lies, damn lies, and statistics”. Perhaps it is time to update the quote attributed to Disraeli. I’d like to suggest, “Lies, pure BS, and polls”..Every few weeks some new poll reports what Canadians or Albertans are thinking. Please don’t be fooled... the intention of this reporting is not to inform you what others are thinking. No, it is intended to change your opinion without you having to do any serious thinking for yourself. So, perhaps if you can spare a few minutes to read further, you might examine the next published polling results with a more critical mind..The methods of applied statistics are well understood and of huge benefits where quantitative measurement is impossible and so sampling a population is required. Take for example the classic problem of estimating the number of black marbles in a jar of white and black marbles. You could empty the jar and count every marble. Or you could take a few dozen out of the jar, count the number of whites and blacks and make a conclusion on the entire population. Statistics will tell you how confident you can be about your conclusions — this is the “+/- 2.8%, 19 times out of 20.” All of this is fine. But where it gets a bit murky is when humans are surveyed about their opinions, and then we try to predict election outcomes, or try to influence government policy formation based on these opinions. This is where you need to be s bit more sceptical of what you are reading..Is counting opinions that much different than counting marbles? What could go wrong? Let me count the ways..The difficulty for pollsters is to frame neutral questions that don’t in the very asking, prejudice the answers. Modern psychological research shows how difficult this can be. Author Malcolm Gladwell in his book 'Blink' explores both the benefits and the dark side of our instantaneous reactions. He explains how not only our attitudes and opinions can be ‘primed’ by words and images, but also our performance. It seems that if you are asked to think about words like ‘professor’, ‘intelligence’, and ‘wisdom’ before you play Trivial Pursuit, you will actually score higher than if you are asked to think about words like ‘soccer hooligans’, ‘stupid’ and ‘fool-hardy’. Psychologists conclude that our unconscious attitudes revealed in a ‘blink’ may be completely incompatible with our stated conscious values. Further, the opinions we profess today may well be due to the priming we heard in the last week or the last year which may have never been filtered by actual critical thinking..If you think the psychologists are of little help, then don’t look to the economists. Like the author of the biblical Book of James, economists think it is not what we believe or say that is important, but rather what we do. The only true preferences are revealed preferences; those preferences revealed by action. As James says, “faith by itself, if it is not accompanied by action, is dead”. My own personal preference in a car is a Jaguar and yet I drive a Toyota. My revealed preference when I actually have to make a real decision with real money in the real world is my actual real preference. This is why the question “If an election were held today ....” doesn’t really get at what I would actually do if there was a real election today. Thus the results may be suspect. But there are other issues as well..First, beware of partisan polls. These are polls that are undertaken to support a particular position. They are generally sponsored by a lobby group that is trying to change your opinion and thus claim public support for their preferred policy..For example, the RCMP union did a poll of Albertans in August 2022 about a move to a provincial police force. Not surprisingly, the pollster, Pollara, found that “Majority of Albertans opposed to provincial police force” to quote the Calgary Herald headline. The survey did acknowledge that more people where the RCMP were actually the police, favoured a move to a provincial police force. “What?” you say? Why were people who have their own police forces e.g. Calgary and Edmonton, asked about the police service that their rural neighbours should use? Why is it any of their business? It is a good question and points to a scientific error of the poll — they surveyed the wrong underlying population. It would have made as much sense to ask the residents of Montana the same question. Further, the Pollara poll contradicts an Angus Reid poll done at the same time that found that overall, people had less confidence in the RCMP than in their local provincial or municipal police force. Both these polls can’t be right. If a poll is partisan it should be suspect..Another example of a recent partisan poll is one done for Clean Energy Canada by Abacus Data. The lobby group wants to influence your thinking towards more “green and clean” energy options. Can you spot any priming going on here? But also, they are asking for people’s beliefs about matters of fact. Despite Albertans electricity costs rising due to the NDP forced phase out of coal fired electrical plants, half of them still think (or at least have been primed to think) that “clean” energy is more affordable. Also despite motor gas bills going up because of insecure global oil supplies, and our electricity grid nearly failing a number of times in sunless, windless December, half of Albertans still think clean is more secure. As the saying goes, there is no solution for insanity — 100% of surveyed people in court felt the emperor's new clothes were gorgeous..Second, beware of polls that survey the wrong underlying population (similar to the (intentional) RCMP union poll above.) This error was most apparent during the recent CPC and UCP leadership contests..The most exceptional thing about Pierre Poilievre’s win is that he obtained nearly 71% (68% of the points required) of the votes. This outstripped his polling numbers by 27-37% points as he was polling in the 34-44% range in the last six weeks before the vote. This is a polling error of 61-108% which is a far cry from “+/- 2.8%,19 times out of 20.” The reason for the error is all Canadians were polled, not just CPC members. But that didn’t prevent the numbers from being made public as if they were meaningful measures of who would become the next leader..The same thing happened to Danielle Smith. The Calgary Herald and Edmonton Journal hired Leger in July 2022 and September 2022 to poll Albertans concerning their voting preferences and their preferences for the leadership of the UCP. In July, Smith’s numbers by self-declared “UCP supporters”, but not necessarily UCP members, were listed as 22% followed by Brian Jean at 20% and Travis Toews at 15%. In September, in an attempt to simulate the ranked ballot voting, the first ballot numbers (again by those self identifying as UCP supporters) were Jean 31%, Toews 29%, and Smith 27%. Despite Don Braid knowing that actual UCP members were not surveyed he fearlessly lead his column “Brian Jean, premier of Alberta?” and said Smith “scores no higher than 31 per cent in any round (on Ballot 5) after starting out at just 27 per cent on the first ballot.”.The actual results on the first ballot were Smith 41.3%, Toews 29.4% and Jean 11.0 % - a 180% error. Again, beware if the survey population isn’t the actual people making the decisions..Third, remember that polls are not predictions. When the pollster asks “If an election were held today...” they are not trying to make a prediction on what will happen in the next election (except they let their readers make that leap). Further, if they were trying to make a prediction then they are asking the wrong question. Isn’t the proper question “When the next provincial election is held on May 29, 2023 what party ...”. Of course the right answer to this question is “Depends on what happens between now and then” But pollsters can’t get any headlines with results like “80% of active voters will wait for the election campaign to decide their votes” or “20% of voters will not be in town on election day” or “35% of Albertans are so disillusioned about our democratic institutions that they will never vote again”. (Remember, over 30% of eligible UCP members did not cast a ballot in what was arguably the most important leadership race since Jason Kenney bested Brian Jean.).If pollsters wanted more accuracy they would ask two important questions on any poll of voting intentions: 1. “Did you vote in the last election”? A ‘No’ answer would result in no further questions. 2. “Based on a scale from 1 to 10 where 1 is not determined at all and 10 is extremely determined... how determined are you to vote in the upcoming election”? Any number less than 6 would result in no further questions. Why would you survey people who do not participate in the democratic process about their political preferences or their policy preferences? Those preferences obviously won’t matter in the actual election. In a recent poll by Abacus Data, a self-titled “Alberta Politics Deep Dive” got detailed views on policy and “impressions” from people, 23% of whom “didn’t vote in the last election and likely won’t vote again”..This type of polling flaw shows up in all sorts of weird results. In the 2019 Alberta provincial election the UCP was consistently polling around 10% points above the NDP. However, when the votes were counted the UCP was 22% points ahead, a 120% polling error. Turns out the people answering the pollsters weren’t the people who actually took the time to vote..So what is person to do if they want to an idea about what might happen if an election were actually held today? Well, they have to dig a little deeper into the realm of polling aggregation and election modelling. In Canada, Eric Grenier provided this in the past but has moved on to be a CBC commentator. His role was taken over by Phillipe J. Fournier, who runs the website 338canada.com..The poll aggregators, as the name implies, put polling data together but weight it by its age and a quality ranking of the pollster. Then they integrate that data amongst other factors to predict election outcomes. The model methods are not secret and are explained on the website. The models work at the riding level but provide probabilistic outcomes for total popular vote, seat projection, the odds (probability) of winning the most seats and the odds of a majority or minority government. This is where you should look if you want to know what would likely happen, “if an election was held today.” Fournier has had pretty accurate results including the 2019 Alberta general election where he predicted 60 seats for the UCP and they won 63. Remember, he used the pollsters data which had more than 100% polling error for that election..So what does Fournier say now about the upcoming election? “Between now and election day, there are lots of variables at play.“ No kidding. But when he runs his models, the “day of” predictions consistently favour a UCP majority. When the model was run on December 9 the odds of a UCP majority was 66% which fell to 60% on December 23 and rose to 68% on January 29. The 6% drop on December 23 was due to the “Deep Dive” Abacus poll which concluded, “This snapshot of Alberta public opinion points to an advantage for the Alberta NDP and Rachel Notley in the early days of Danielle Smith’s time in office. The Alberta NDP has a lead in vote intentions, the more popular leader, and issues ownership on more of the top issues for voters.”.Sounds a bit partisan to me — are you convinced?.Part of Smith’s strategy is to try and win over voters who may be hesitant in accepting her re-birth in politics. She has somewhat less than four more months to work her magic..In the meantime, ignore the polls but occasionally look to see what Fournier is saying..William D. Marriot is a retired economist who studied way too much statistics and econometrics while at university.
