When good causes use bad stats

Get your science fix here: research, quackery, activism and all the rest
Post Reply
User avatar
Tessa K
Light of Blast
Posts: 4707
Joined: Tue Nov 12, 2019 5:07 pm
Location: Closer than you'd like

When good causes use bad stats

Post by Tessa K » Thu Dec 05, 2019 11:06 am

Here's a quick short piece I wrote on when good causes use bad stats (Note: I am not a statistician).

[mod]Thread title and post edited to make the title more descriptive of the contents[/mod]

User avatar
JQH
After Pie
Posts: 2142
Joined: Mon Nov 11, 2019 3:30 pm
Location: Sar Flandan

Re: When good causes use bad stats

Post by JQH » Thu Dec 05, 2019 12:36 pm

It's a mistake to use bad stats in a good cause - you finish up with a lot of argument about the numbers and the substantive -"Let's reduce road deaths" in this case - gets lost. It would have been easy enough to make it clear that the 3,500 per day is a global figure.

Good blog post btw.
And remember that if you botch the exit, the carnival of reaction may be coming to a town near you.

Fintan O'Toole

User avatar
Tessa K
Light of Blast
Posts: 4707
Joined: Tue Nov 12, 2019 5:07 pm
Location: Closer than you'd like

Re: When good causes use bad stats

Post by Tessa K » Thu Dec 05, 2019 1:13 pm

JQH wrote:
Thu Dec 05, 2019 12:36 pm
It's a mistake to use bad stats in a good cause - you finish up with a lot of argument about the numbers and the substantive -"Let's reduce road deaths" in this case - gets lost. It would have been easy enough to make it clear that the 3,500 per day is a global figure.

Good blog post btw.
Thanks.

Both the WHO and Wiki have handy tables of stats. It's not like I spent hours trying to source them. I did actually make a mistake with the UK figures that I've now changed. It's a surprisingly low figure for us - but the injury figures are pretty high.

It's clear the campaign isn't aimed at me at all as I had no idea who the footballer was or the driver.

tom p
After Pie
Posts: 1876
Joined: Mon Nov 11, 2019 1:14 pm
Location: the low countries

Re: When good causes use bad stats

Post by tom p » Thu Dec 05, 2019 2:30 pm

It was indeed a good blogpost.

Charles Leclerc (the F1 racing driver who drives for Ferrari and is going to be the best driver in the world in a couple of years When Hamilton (Lewis, not the musical) retires) is from Monaco, so using the Monegasque princess with him kinda makes sense. Although I doubt they have many posters up in Monaco, since the cost:benefit ratio would be pretty small.

The footballer (Griezmann) is one of the best and most famous in the world - he's on the rung just below the likes of Messi & Ronaldo that everyone has heard of, and on the level where everyone who like football has heard of him (and wishes he played for their team).

I have to say, though, that the FIA is an international organisation & it's a worldwide campaign, so using global figures isn't really misleading. They don't say 'today 3,500 people will die on your roads' or ...'on the UK's roads'. It's a true statement, even though addition of the word "worldwide" would have been clearer and more transparent.

It was definitely amusing to see the FIA referred to as the good guys, that's for sure!

User avatar
Tessa K
Light of Blast
Posts: 4707
Joined: Tue Nov 12, 2019 5:07 pm
Location: Closer than you'd like

Re: When good causes use bad stats

Post by Tessa K » Thu Dec 05, 2019 3:08 pm

tom p wrote:
Thu Dec 05, 2019 2:30 pm
It was indeed a good blogpost.

Charles Leclerc (the F1 racing driver who drives for Ferrari and is going to be the best driver in the world in a couple of years When Hamilton (Lewis, not the musical) retires) is from Monaco, so using the Monegasque princess with him kinda makes sense. Although I doubt they have many posters up in Monaco, since the cost:benefit ratio would be pretty small.

The footballer (Griezmann) is one of the best and most famous in the world - he's on the rung just below the likes of Messi & Ronaldo that everyone has heard of, and on the level where everyone who like football has heard of him (and wishes he played for their team).

I have to say, though, that the FIA is an international organisation & it's a worldwide campaign, so using global figures isn't really misleading. They don't say 'today 3,500 people will die on your roads' or ...'on the UK's roads'. It's a true statement, even though addition of the word "worldwide" would have been clearer and more transparent.

It was definitely amusing to see the FIA referred to as the good guys, that's for sure!
Monaco has the lowest road deaths in the world per capita so unless the princess likes that photo of her and has persuaded them to put up a few, I doubt the campaign is running there.

You're right, it is a global campaign which is why I said 'misleading' not 'liar liar pants on fire' - but stats always need context to have meaning. Even if they said 'worldwide' on the posters it would still be better to know how many in this country. It's not like they have to physically print and deliver the posters any more so the cost of making them country specific is minimal. Nearly five a day is still a lot - maybe not if you're a young bloke who likes a bit of risk. Do you feel lucky? Well, do you? :D

User avatar
Gfamily
Light of Blast
Posts: 5180
Joined: Mon Nov 11, 2019 1:00 pm
Location: NW England

Re: When good causes use bad stats

Post by Gfamily » Thu Dec 05, 2019 3:54 pm

Is it nit-picking to point out that although the headline message of the poster is to watch out for pedestrians, they comprise only about 1/4 of the total road deaths?
Of course it is. But it allows for that sort of nit-picking, so in this context it's probably valid.
My avatar was a scientific result that was later found to be 'mistaken' - I rarely claim to be 100% correct
ETA 5/8/20: I've been advised that the result was correct, it was the initial interpretation that needed to be withdrawn
Meta? I'd say so!

User avatar
Tessa K
Light of Blast
Posts: 4707
Joined: Tue Nov 12, 2019 5:07 pm
Location: Closer than you'd like

Re: When good causes use bad stats

Post by Tessa K » Thu Dec 05, 2019 6:16 pm

Gfamily wrote:
Thu Dec 05, 2019 3:54 pm
Is it nit-picking to point out that although the headline message of the poster is to watch out for pedestrians, they comprise only about 1/4 of the total road deaths?
Of course it is. But it allows for that sort of nit-picking, so in this context it's probably valid.
As part of the package of posters promoting road safety in general, I guess it's fair enough. As a pedestrian, I'm all in favour.

User avatar
Martin Y
Stummy Beige
Posts: 3080
Joined: Mon Nov 11, 2019 1:08 pm

Re: When good causes use bad stats

Post by Martin Y » Thu Dec 05, 2019 6:34 pm

The Monaco royals always make me think of princess Grace (former Hollywood star Grace Kelly, of course) who famously died from a car crash. I wonder if the demise of her would-have-been mother in law in the '80s could be why princess Charlene is in the poster.

Mind you it's so long ago it would surely be a reference entirely lost on most of the posters' intended audience.

User avatar
snoozeofreason
Snowbonk
Posts: 485
Joined: Fri Nov 15, 2019 1:22 pm

Re: When good causes use bad stats

Post by snoozeofreason » Thu Dec 05, 2019 6:35 pm

Excellent blog post! I am afraid that the problem is ubiquitous. Pretty much all good causes use bad stats, at least that's the way it appears to me. Maybe the desire to change things doesn't often coincide with a desire to present them in a numerically accurate way. It's hard to know what to do about it. Attempts to correct the bad stats generally don't go down well, but it they aren't corrected then all sorts of unintended consequences can follow. If you have been given an inflated figure for road traffic accidents or something, then any more realistic figure will create an impression that the problem is getting better when it may well be getting worse.
In six days the Lord made the heavens and the earth, the sea, and all that is in them. The human body was knocked up pretty late on the Friday afternoon, with a deadline looming. How well do you expect it to work?

User avatar
Sciolus
Dorkwood
Posts: 1314
Joined: Mon Nov 11, 2019 6:42 pm

Re: When good causes use bad stats

Post by Sciolus » Thu Dec 05, 2019 8:17 pm

JQH wrote:
Thu Dec 05, 2019 12:36 pm
It's a mistake to use bad stats in a good cause - you finish up with a lot of argument about the numbers and the substantive -"Let's reduce road deaths" in this case - gets lost. It would have been easy enough to make it clear that the 3,500 per day is a global figure.
By putting the words "global road safety" on it for instance?

ControlFreak
Ghastly Pink
Posts: 4
Joined: Sun Dec 01, 2019 3:31 pm

Re: When good causes use bad stats

Post by ControlFreak » Thu Dec 05, 2019 8:33 pm

I agree that using stats for a blazing headline without qualifying the scope and source takes away from the credibility of the cause. I 'll have to admit that I assumed it was the global figure but that was because know the uk figure as I often use it to help give people perspective when doing risk subjective risk analysis.

I also came across some thing similar today after seeing the bbc news report (of a Sutton Trust report) comparing new gcse results for advantaged vs disadvantaged students. They made a load of conclusions based on models with R^2 of less than 0.3 in most cases!

User avatar
Bird on a Fire
Princess POW
Posts: 10137
Joined: Fri Oct 11, 2019 5:05 pm
Location: Portugal

Re: When good causes use bad stats

Post by Bird on a Fire » Thu Dec 05, 2019 9:14 pm

ControlFreak wrote:
Thu Dec 05, 2019 8:33 pm
I also came across some thing similar today after seeing the bbc news report (of a Sutton Trust report) comparing new gcse results for advantaged vs disadvantaged students. They made a load of conclusions based on models with R^2 of less than 0.3 in most cases!
Honestly explaining 30% of variance is pretty decent in messy fields like sociology and parts of the life sciences. Obviously there will be a huge number of influences on exam performance - if you can predict successfully using basic demographic data that's impressive and useful. And if even 10% of exam result variation is predicted by social background I'd argue that's important, and should motivate political action.
We have the right to a clean, healthy, sustainable environment.

User avatar
snoozeofreason
Snowbonk
Posts: 485
Joined: Fri Nov 15, 2019 1:22 pm

Re: When good causes use bad stats

Post by snoozeofreason » Thu Dec 05, 2019 10:25 pm

I suspect that it is going to be so easy to find examples of good causes using bad stats that this could turn into a megathread, but here is one that annoyed me from the Guardian earlier this week.
Brexit is contributing to a serious brain drain in UK universities, say the Liberal Democrats, after it emerged that almost 11,000 EU academics had left since the 2016 referendum.

The figures, based on freedom of information responses from universities, show 10,918 left in the three years starting with the 2016-17 financial year. In 2018-19, 4,014 quit, 31% more than in 2015-16, and 40% more than in 2014-15.
I am an academic, and lots of my colleagues are from other* EU countries, so I completely agree that we should worry about Brexit driving valuable staff away. But even in the initial two paragraphs that I have quoted there are enough questions to annoy any academic. For example:
  • What are they counting? Is it EU27 academics who have left the UK, or simply those who have left a university in the UK, possibly to join another university in the UK. The context suggests that it might be the latter.
  • How many EU27 academics would you expect to leave in the period in question? After all academe is global, and academics are often quite mobile, particularly at the start of the careers.
  • Why does the last sentence refer to figures from single years, when the previous sentences referred to a total over a three year period?
* I would guess that where the article refers to "11,000 EU academics" the Guardian has temporarily forgotten that the UK is still in the EU, and that therefore all British staff are EU academics.
In six days the Lord made the heavens and the earth, the sea, and all that is in them. The human body was knocked up pretty late on the Friday afternoon, with a deadline looming. How well do you expect it to work?

cvb
Clardic Fug
Posts: 228
Joined: Tue Nov 12, 2019 8:15 am

Re: When good causes use bad stats

Post by cvb » Fri Dec 06, 2019 9:00 am

snoozeofreason wrote:
Thu Dec 05, 2019 10:25 pm
I suspect that it is going to be so easy to find examples of good causes using bad stats that this could turn into a megathread, but here is one that annoyed me from the Guardian earlier this week.
Brexit is contributing to a serious brain drain in UK universities, say the Liberal Democrats, after it emerged that almost 11,000 EU academics had left since the 2016 referendum.

The figures, based on freedom of information responses from universities, show 10,918 left in the three years starting with the 2016-17 financial year. In 2018-19, 4,014 quit, 31% more than in 2015-16, and 40% more than in 2014-15.
I am an academic, and lots of my colleagues are from other* EU countries, so I completely agree that we should worry about Brexit driving valuable staff away. But even in the initial two paragraphs that I have quoted there are enough questions to annoy any academic. For example:
  • What are they counting? Is it EU27 academics who have left the UK, or simply those who have left a university in the UK, possibly to join another university in the UK. The context suggests that it might be the latter.
  • How many EU27 academics would you expect to leave in the period in question? After all academe is global, and academics are often quite mobile, particularly at the start of the careers.
  • Why does the last sentence refer to figures from single years, when the previous sentences referred to a total over a three year period?
* I would guess that where the article refers to "11,000 EU academics" the Guardian has temporarily forgotten that the UK is still in the EU, and that therefore all British staff are EU academics.
From the article.
The Lib Dem statistics only cover departures, and not EU nationals entering the UK to take up academic posts. Figures from the Higher Education Statistics Agency show that the overall number of EU academic staff rose slightly between 2016-17 and 2017-18, the most recent data available, from 35,920 to 37,255.

User avatar
snoozeofreason
Snowbonk
Posts: 485
Joined: Fri Nov 15, 2019 1:22 pm

Re: When good causes use bad stats

Post by snoozeofreason » Fri Dec 06, 2019 1:28 pm

Also from the article:
In September, Universities UK said a survey of members had found that almost 60% had lost existing or potential staff to overseas institutions;
The figure of 60% seems bizarrely low, given that a typical university employs a lot of people, and that academics are quite internationally mobile. Can it really be true that in over 40% of the member universities no one at all left for an overseas institution? The inclusion of "potential staff" makes it even more odd. At a guess I'd imagine there were a lot of "don't know"s in the response to that question, but either way it's hard to see what the point of quoting the statistic is.
In six days the Lord made the heavens and the earth, the sea, and all that is in them. The human body was knocked up pretty late on the Friday afternoon, with a deadline looming. How well do you expect it to work?

Bewildered
Fuzzable
Posts: 257
Joined: Tue Nov 12, 2019 2:51 pm

Re: When good causes use bad stats

Post by Bewildered » Sat Dec 07, 2019 10:00 pm

snoozeofreason wrote:
Fri Dec 06, 2019 1:28 pm
Also from the article:
In September, Universities UK said a survey of members had found that almost 60% had lost existing or potential staff to overseas institutions;
The figure of 60% seems bizarrely low, given that a typical university employs a lot of people, and that academics are quite internationally mobile. Can it really be true that in over 40% of the member universities no one at all left for an overseas institution? The inclusion of "potential staff" makes it even more odd. At a guess I'd imagine there were a lot of "don't know"s in the response to that question, but either way it's hard to see what the point of quoting the statistic is.
Sufficiently unlikely that i’d say it’s effectively impossible, but I guess it depends on the exact question and how it was interpreted. The respondents, or enough of them, may have interpreted it as asking if they losing more people than normal or spevificslly as a result of brexit, which is what the journalist probably thinks it means even though the phrase they used doesn’t. Without the actual survey question it is hard to distinguish poor reporting from a bad survey question or misuse of a good one etc.

User avatar
snoozeofreason
Snowbonk
Posts: 485
Joined: Fri Nov 15, 2019 1:22 pm

Re: When good causes use bad stats

Post by snoozeofreason » Mon Dec 09, 2019 3:10 pm

Bewildered wrote:
Sat Dec 07, 2019 10:00 pm
Sufficiently unlikely that i’d say it’s effectively impossible, but I guess it depends on the exact question and how it was interpreted. The respondents, or enough of them, may have interpreted it as asking if they losing more people than normal or spevificslly as a result of brexit, which is what the journalist probably thinks it means even though the phrase they used doesn’t. Without the actual survey question it is hard to distinguish poor reporting from a bad survey question or misuse of a good one etc.
Another possibility is that the respondents were saying whether they knew of staff who had left for an overseas institution. When you quit a job you don't necessarily leave any official record of where you move on to, so maybe the respondents didn't have had the information that was needed to answer the question properly.
In six days the Lord made the heavens and the earth, the sea, and all that is in them. The human body was knocked up pretty late on the Friday afternoon, with a deadline looming. How well do you expect it to work?

Post Reply