Page 1 of 1
When good causes use bad stats
Posted: Thu Dec 05, 2019 11:06 am
by Tessa K
Here's a
quick short piece I wrote on when good causes use bad stats (Note: I am not a statistician).
[mod]Thread title and post edited to make the title more descriptive of the contents[/mod]
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 12:36 pm
by JQH
It's a mistake to use bad stats in a good cause - you finish up with a lot of argument about the numbers and the substantive -"Let's reduce road deaths" in this case - gets lost. It would have been easy enough to make it clear that the 3,500 per day is a global figure.
Good blog post btw.
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 1:13 pm
by Tessa K
JQH wrote: ↑Thu Dec 05, 2019 12:36 pm
It's a mistake to use bad stats in a good cause - you finish up with a lot of argument about the numbers and the substantive -"Let's reduce road deaths" in this case - gets lost. It would have been easy enough to make it clear that the 3,500 per day is a global figure.
Good blog post btw.
Thanks.
Both the WHO and Wiki have handy tables of stats. It's not like I spent hours trying to source them. I did actually make a mistake with the UK figures that I've now changed. It's a surprisingly low figure for us - but the injury figures are pretty high.
It's clear the campaign isn't aimed at me at all as I had no idea who the footballer was or the driver.
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 2:30 pm
by tom p
It was indeed a good blogpost.
Charles Leclerc (the F1 racing driver who drives for Ferrari and is going to be the best driver in the world in a couple of years When Hamilton (Lewis, not the musical) retires) is from Monaco, so using the Monegasque princess with him kinda makes sense. Although I doubt they have many posters up in Monaco, since the cost:benefit ratio would be pretty small.
The footballer (Griezmann) is one of the best and most famous in the world - he's on the rung just below the likes of Messi & Ronaldo that everyone has heard of, and on the level where everyone who like football has heard of him (and wishes he played for their team).
I have to say, though, that the FIA is an international organisation & it's a worldwide campaign, so using global figures isn't really misleading. They don't say 'today 3,500 people will die on your roads' or ...'on the UK's roads'. It's a true statement, even though addition of the word "worldwide" would have been clearer and more transparent.
It was definitely amusing to see the FIA referred to as the good guys, that's for sure!
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 3:08 pm
by Tessa K
tom p wrote: ↑Thu Dec 05, 2019 2:30 pm
It was indeed a good blogpost.
Charles Leclerc (the F1 racing driver who drives for Ferrari and is going to be the best driver in the world in a couple of years When Hamilton (Lewis, not the musical) retires) is from Monaco, so using the Monegasque princess with him kinda makes sense. Although I doubt they have many posters up in Monaco, since the cost:benefit ratio would be pretty small.
The footballer (Griezmann) is one of the best and most famous in the world - he's on the rung just below the likes of Messi & Ronaldo that everyone has heard of, and on the level where everyone who like football has heard of him (and wishes he played for their team).
I have to say, though, that the FIA is an international organisation & it's a worldwide campaign, so using global figures isn't really misleading. They don't say 'today 3,500 people will die on your roads' or ...'on the UK's roads'. It's a true statement, even though addition of the word "worldwide" would have been clearer and more transparent.
It was definitely amusing to see the FIA referred to as the good guys, that's for sure!
Monaco has the lowest road deaths in the world
per capita so unless the princess likes that photo of her and has persuaded them to put up a few, I doubt the campaign is running there.
You're right, it is a global campaign which is why I said 'misleading' not 'liar liar pants on fire' - but stats always need context to have meaning. Even if they said 'worldwide' on the posters it would still be better to know how many in this country. It's not like they have to physically print and deliver the posters any more so the cost of making them country specific is minimal. Nearly five a day is still a lot - maybe not if you're a young bloke who likes a bit of risk. Do you feel lucky? Well, do you?
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 3:54 pm
by Gfamily
Is it nit-picking to point out that although the headline message of the poster is to watch out for pedestrians, they comprise only about 1/4 of the total road deaths?
Of course it is. But it allows for that sort of nit-picking, so in this context it's probably valid.
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 6:16 pm
by Tessa K
Gfamily wrote: ↑Thu Dec 05, 2019 3:54 pm
Is it nit-picking to point out that although the headline message of the poster is to watch out for pedestrians, they comprise only about 1/4 of the total road deaths?
Of course it is. But it allows for that sort of nit-picking, so in
this context it's probably valid.
As part of the package of posters promoting road safety in general, I guess it's fair enough. As a pedestrian, I'm all in favour.
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 6:34 pm
by Martin Y
The Monaco royals always make me think of princess Grace (former Hollywood star Grace Kelly, of course) who famously died from a car crash. I wonder if the demise of her would-have-been mother in law in the '80s could be why princess Charlene is in the poster.
Mind you it's so long ago it would surely be a reference entirely lost on most of the posters' intended audience.
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 6:35 pm
by snoozeofreason
Excellent blog post! I am afraid that the problem is ubiquitous. Pretty much all good causes use bad stats, at least that's the way it appears to me. Maybe the desire to change things doesn't often coincide with a desire to present them in a numerically accurate way. It's hard to know what to do about it. Attempts to correct the bad stats generally don't go down well, but it they aren't corrected then all sorts of unintended consequences can follow. If you have been given an inflated figure for road traffic accidents or something, then any more realistic figure will create an impression that the problem is getting better when it may well be getting worse.
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 8:17 pm
by Sciolus
JQH wrote: ↑Thu Dec 05, 2019 12:36 pm
It's a mistake to use bad stats in a good cause - you finish up with a lot of argument about the numbers and the substantive -"Let's reduce road deaths" in this case - gets lost. It would have been easy enough to make it clear that the 3,500 per day is a global figure.
By putting the words "global road safety" on it for instance?
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 8:33 pm
by ControlFreak
I agree that using stats for a blazing headline without qualifying the scope and source takes away from the credibility of the cause. I 'll have to admit that I assumed it was the global figure but that was because know the uk figure as I often use it to help give people perspective when doing risk subjective risk analysis.
I also came across some thing similar today after seeing the bbc news report (of a Sutton Trust report) comparing new gcse results for advantaged vs disadvantaged students. They made a load of conclusions based on models with R^2 of less than 0.3 in most cases!
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 9:14 pm
by Bird on a Fire
ControlFreak wrote: ↑Thu Dec 05, 2019 8:33 pm
I also came across some thing similar today after seeing the bbc news report (of a Sutton Trust report) comparing new gcse results for advantaged vs disadvantaged students. They made a load of conclusions based on models with R^2 of less than 0.3 in most cases!
Honestly explaining 30% of variance is pretty decent in messy fields like sociology and parts of the life sciences. Obviously there will be a huge number of influences on exam performance - if you can predict successfully using basic demographic data that's impressive and useful. And if even 10% of exam result variation is predicted by social background I'd argue that's important, and should motivate political action.
Re: When good causes use bad stats
Posted: Thu Dec 05, 2019 10:25 pm
by snoozeofreason
I suspect that it is going to be so easy to find examples of good causes using bad stats that this could turn into a megathread, but
here is one that annoyed me from the Guardian earlier this week.
Brexit is contributing to a serious brain drain in UK universities, say the Liberal Democrats, after it emerged that almost 11,000 EU academics had left since the 2016 referendum.
The figures, based on freedom of information responses from universities, show 10,918 left in the three years starting with the 2016-17 financial year. In 2018-19, 4,014 quit, 31% more than in 2015-16, and 40% more than in 2014-15.
I am an academic, and lots of my colleagues are from other* EU countries, so I completely agree that we should worry about Brexit driving valuable staff away. But even in the initial two paragraphs that I have quoted there are enough questions to annoy any academic. For example:
- What are they counting? Is it EU27 academics who have left the UK, or simply those who have left a university in the UK, possibly to join another university in the UK. The context suggests that it might be the latter.
- How many EU27 academics would you expect to leave in the period in question? After all academe is global, and academics are often quite mobile, particularly at the start of the careers.
- Why does the last sentence refer to figures from single years, when the previous sentences referred to a total over a three year period?
* I would guess that where the article refers to "11,000 EU academics" the Guardian has temporarily forgotten that the UK is still in the EU, and that therefore all British staff are EU academics.
Re: When good causes use bad stats
Posted: Fri Dec 06, 2019 9:00 am
by cvb
snoozeofreason wrote: ↑Thu Dec 05, 2019 10:25 pm
I suspect that it is going to be so easy to find examples of good causes using bad stats that this could turn into a megathread, but
here is one that annoyed me from the Guardian earlier this week.
Brexit is contributing to a serious brain drain in UK universities, say the Liberal Democrats, after it emerged that almost 11,000 EU academics had left since the 2016 referendum.
The figures, based on freedom of information responses from universities, show 10,918 left in the three years starting with the 2016-17 financial year. In 2018-19, 4,014 quit, 31% more than in 2015-16, and 40% more than in 2014-15.
I am an academic, and lots of my colleagues are from other* EU countries, so I completely agree that we should worry about Brexit driving valuable staff away. But even in the initial two paragraphs that I have quoted there are enough questions to annoy any academic. For example:
- What are they counting? Is it EU27 academics who have left the UK, or simply those who have left a university in the UK, possibly to join another university in the UK. The context suggests that it might be the latter.
- How many EU27 academics would you expect to leave in the period in question? After all academe is global, and academics are often quite mobile, particularly at the start of the careers.
- Why does the last sentence refer to figures from single years, when the previous sentences referred to a total over a three year period?
* I would guess that where the article refers to "11,000 EU academics" the Guardian has temporarily forgotten that the UK is still in the EU, and that therefore all British staff are EU academics.
From the article.
The Lib Dem statistics only cover departures, and not EU nationals entering the UK to take up academic posts. Figures from the Higher Education Statistics Agency show that the overall number of EU academic staff rose slightly between 2016-17 and 2017-18, the most recent data available, from 35,920 to 37,255.
Re: When good causes use bad stats
Posted: Fri Dec 06, 2019 1:28 pm
by snoozeofreason
Also from the article:
In September, Universities UK said a survey of members had found that almost 60% had lost existing or potential staff to overseas institutions;
The figure of 60% seems bizarrely low, given that a typical university employs a lot of people, and that academics are quite internationally mobile. Can it really be true that in over 40% of the member universities no one at all left for an overseas institution? The inclusion of "potential staff" makes it even more odd. At a guess I'd imagine there were a lot of "don't know"s in the response to that question, but either way it's hard to see what the point of quoting the statistic is.
Re: When good causes use bad stats
Posted: Sat Dec 07, 2019 10:00 pm
by Bewildered
snoozeofreason wrote: ↑Fri Dec 06, 2019 1:28 pm
Also from the article:
In September, Universities UK said a survey of members had found that almost 60% had lost existing or potential staff to overseas institutions;
The figure of 60% seems bizarrely low, given that a typical university employs a lot of people, and that academics are quite internationally mobile. Can it really be true that in over 40% of the member universities no one at all left for an overseas institution? The inclusion of "potential staff" makes it even more odd. At a guess I'd imagine there were a lot of "don't know"s in the response to that question, but either way it's hard to see what the point of quoting the statistic is.
Sufficiently unlikely that i’d say it’s effectively impossible, but I guess it depends on the exact question and how it was interpreted. The respondents, or enough of them, may have interpreted it as asking if they losing more people than normal or spevificslly as a result of brexit, which is what the journalist probably thinks it means even though the phrase they used doesn’t. Without the actual survey question it is hard to distinguish poor reporting from a bad survey question or misuse of a good one etc.
Re: When good causes use bad stats
Posted: Mon Dec 09, 2019 3:10 pm
by snoozeofreason
Bewildered wrote: ↑Sat Dec 07, 2019 10:00 pm
Sufficiently unlikely that i’d say it’s effectively impossible, but I guess it depends on the exact question and how it was interpreted. The respondents, or enough of them, may have interpreted it as asking if they losing more people than normal or spevificslly as a result of brexit, which is what the journalist probably thinks it means even though the phrase they used doesn’t. Without the actual survey question it is hard to distinguish poor reporting from a bad survey question or misuse of a good one etc.
Another possibility is that the respondents were saying whether they
knew of staff who had left for an overseas institution. When you quit a job you don't necessarily leave any official record of where you move on to, so maybe the respondents didn't have had the information that was needed to answer the question properly.