Survivor Bias

[Tessa K]

There's a lot of survivor bias going on. I was aware people were doing this but only recently learnt there's a name for it.

I got it and I'm fine so it's not that bad. My neighbour's auntie's cousin is 80 and survived so it's not that bad. I don't care if I get it because I don't know anyone who got it and died...

https://sites.google.com/site/skeptica ... rship-bias

[bjn]

The first time I’d heard of survivorship bias was in relation to WWII bombers and where to put armour plating. The USAF had to figure out where to put heavy armour plating on their planes as they were being shot down over Europe in intolerable numbers. However you can’t put it everywhere as it’s too heavy, so they counted the bullet holes in the returning planes, and put the armour in the places where those planes didn’t show damage. The statistician doing the analysis figured that the entire sample needed to include the planes that didn’t return. There were far fewer bullet holes in tail, cockpit and engines of the surviving planes, implying that it was there that the downed planes were most likely hit and destroyed. So that’s where they put the armour and far more planes ended up surviving.

[Bird on a Fire]

That statistician was Abraham Wald, who went on to do some pretty important stuff. https://en.wikipedia.org/wiki/Abraham_Wald

[sTeamTraen]

Survivorship bias is everywhere. My favourite is "Eee, they don't make cars like they used to, look at this 1966 Morris Minor, still running, they built cars to last in them days", when of course most cars made in 1966 were completely f.cked by rust or terminal mechanical problems after 7-8 years. And ditto for almost any other consumer product.

I would argue that regression to the mean is the continuous variable version, and that's everywhere too.

[Bird on a Fire]

It's a problem in conservation too - so much of what we know about species' ecology comes from studies in highly modified ecosystems. We often have no idea what a species "optimum" might be because those conditions scarcely exist or are drastically under-studied. We also have far more info on common species than rare ones, and of species that can tolerate human-modified systems compared to wilder ones, and way more info from richer/"economically developed" countries despite their comparative paucity of biodiversity.

[JQH]

My favourite examples of the genre are the people who post on Facebook about all the risky things they used to do before elf'n'safety gorn mad and it didn't do them any harm, completely ignoring the fact that those killed doing it can't post on facebook to say so.

[shpalman]

I notice a subtle but related effect in my own field, that people only do in-depth measurements on samples which "work" and then say "look it works because of <result of measurement>".

Maybe the samples which didn't work would look exactly the same in terms of that measurement, but of course you don't know, because you didn't look.

[Little waster]

Though the counter-example is my uncle who survived COVID after spending time in ICU, has been readmitted on several occasions since with post-COVID complications and is now facing a lifetime of COVID-induced CV issues because of it.

He has since set up a blog and has flooded his FB feed with his story as an example how bad it can get as a warning to others.

The irony being of course he was the sort of Elf of Safety gorn mad, Brexit-supporting gammon type who normally would be at the forefront of COVID-scepticism which means his social media connections are all of a similar ilk so his message is particularly impactful to them.

[Tessa K]

The first time I’d heard of survivorship bias was in relation to WWII bombers and where to put armour plating. The USAF had to figure out where to put heavy armour plating on their planes as they were being shot down over Europe in intolerable numbers. However you can’t put it everywhere as it’s too heavy, so they counted the bullet holes in the returning planes, and put the armour in the places where those planes didn’t show damage. The statistician doing the analysis figured that the entire sample needed to include the planes that didn’t return. There were far fewer bullet holes in tail, cockpit and engines of the surviving planes, implying that it was there that the downed planes were most likely hit and destroyed. So that’s where they put the armour and far more planes ended up surviving.

How did they get access to downed planes to compare?

[shpalman]

The first time I’d heard of survivorship bias was in relation to WWII bombers and where to put armour plating. The USAF had to figure out where to put heavy armour plating on their planes as they were being shot down over Europe in intolerable numbers. However you can’t put it everywhere as it’s too heavy, so they counted the bullet holes in the returning planes, and put the armour in the places where those planes didn’t show damage. The statistician doing the analysis figured that the entire sample needed to include the planes that didn’t return. There were far fewer bullet holes in tail, cockpit and engines of the surviving planes, implying that it was there that the downed planes were most likely hit and destroyed. So that’s where they put the armour and far more planes ended up surviving.

How did they get access to downed planes to compare?

They didn't, but that's the point.

It's the difference between "the planes that come back are all damaged in particular spots so let's reinforce those spots" and "the planes that come back are all damaged in particular spots so those spots obviously don't matter so much and let's reinforce the other bits instead".

[shpalman]

Another one, kind of the opposite effect, was when they wanted to know if launching the Space Shuttle when it was a very cold morning would lead to failures of the O-rings on the boosters. But they plotted the number of O-ring failures (I assume they were able to assess one or two minor non-catastrophic failures found when examining the recovered rockets) against air temperature and didn't notice any correlation.

What they should have done was also include on the plot all the times the O-rings didn't fail, which were all at higher ambient temperatures.

[Tessa K]

The first time I’d heard of survivorship bias was in relation to WWII bombers and where to put armour plating. The USAF had to figure out where to put heavy armour plating on their planes as they were being shot down over Europe in intolerable numbers. However you can’t put it everywhere as it’s too heavy, so they counted the bullet holes in the returning planes, and put the armour in the places where those planes didn’t show damage. The statistician doing the analysis figured that the entire sample needed to include the planes that didn’t return. There were far fewer bullet holes in tail, cockpit and engines of the surviving planes, implying that it was there that the downed planes were most likely hit and destroyed. So that’s where they put the armour and far more planes ended up surviving.

How did they get access to downed planes to compare?

They didn't, but that's the point.

It's the difference between "the planes that come back are all damaged in particular spots so let's reinforce those spots" and "the planes that come back are all damaged in particular spots so those spots obviously don't matter so much and let's reinforce the other bits instead".

There were far fewer bullet holes in tail, cockpit and engines of the surviving planes

How did they know there were fewer? I think this is just about choice of words but it is relevant (to me, anyway)

[shpalman]

There were far fewer bullet holes in tail, cockpit and engines of the surviving planes

How did they know there were fewer? I think this is just about choice of words but it is relevant (to me, anyway)

There were far fewer bullet holes in tail, cockpit and engines of the surviving planes compared to how many bullet holes those same surviving planes had in other bits.

near-missing.jpeg (47.84 KiB) Viewed 2635 times

[bjn]

tx shpallers for answering on my behalf. i’m also sorry i wasn’t clearer in the first place.