Fake science playing into the hands of deniers
- sTeamTraen
- After Pie
- Posts: 2558
- Joined: Mon Nov 11, 2019 4:24 pm
- Location: Palma de Mallorca, Spain
Fake science playing into the hands of deniers
You may remember the fake "fish are hoovering up microplastics" article from 3-4 years ago. It got quite a lot of media coverage, as did the investigation into it, and its subsequent retraction. A Swedish scientist claimed to have collected some data that she couldn't have, then when challenged we found that the only copy of the data was on a laptop that had just been stolen and "the backups were lost in a server glitch". (Seriously, if you take public money for research and you actually genuinely lose the data, you should just pay the money back from your own pocket. But of course, there never were any data to be lost.)
Today this article has appeared in Science (the news bit, not the research bit). It shows that there has been extensive fabrication or manipulation of data in many other studies of fish ecology at the lab at James Cook University (JCU) in Australia where Oona Lönnstedt, lead author of the fake microplastics study, got her PhD.
In particular, there have been multiple papers published claiming that ocean acidification due to dissolved CO2 has a huge effect on the behaviour of reef-dwelling fish. Importantly, the claims were mostly about the effect of the CO2 itself, not the effect of warmer water due to carbon emissions. There are at least 20 papers under scrutiny, of which the most prominent is Dixson et al. 2014, which made the front cover of Science in the week it was published.
I was involved in some of the analyses, although Timothy Clark and team did most of the grunt work, including making step-by-step videos to show what they found (I don't have permission to share those at this point). I have blogged briefly about my involvement here.
This is a f.cking disaster, not least because it plays straight into the hands of climate "skeptics". When the Dixson et al. paper appeared, "Watts Up With That?" took it apart for being implausible, and guess what, they were right. Then in 2020 Clark et al. published an article saying how they had been unable to replicate any of the results from the JCU lab, and James Delingpole picked up on it. With this news, he will probably be w.nking himself into a coma.
Clark et al. are not climate "skeptics". They are honest researchers studying the real effects of global warming on marine ecology. Shortly after their paper was published in January 2020, the #pruittdata story (in which a "rising star" researcher in spider ecology was caught making up tens of datasets) broke. That was when Clark and colleagues started digging into the original data from the JCU studies and finding problems. They are very pissed off about this. We all should be.
Today this article has appeared in Science (the news bit, not the research bit). It shows that there has been extensive fabrication or manipulation of data in many other studies of fish ecology at the lab at James Cook University (JCU) in Australia where Oona Lönnstedt, lead author of the fake microplastics study, got her PhD.
In particular, there have been multiple papers published claiming that ocean acidification due to dissolved CO2 has a huge effect on the behaviour of reef-dwelling fish. Importantly, the claims were mostly about the effect of the CO2 itself, not the effect of warmer water due to carbon emissions. There are at least 20 papers under scrutiny, of which the most prominent is Dixson et al. 2014, which made the front cover of Science in the week it was published.
I was involved in some of the analyses, although Timothy Clark and team did most of the grunt work, including making step-by-step videos to show what they found (I don't have permission to share those at this point). I have blogged briefly about my involvement here.
This is a f.cking disaster, not least because it plays straight into the hands of climate "skeptics". When the Dixson et al. paper appeared, "Watts Up With That?" took it apart for being implausible, and guess what, they were right. Then in 2020 Clark et al. published an article saying how they had been unable to replicate any of the results from the JCU lab, and James Delingpole picked up on it. With this news, he will probably be w.nking himself into a coma.
Clark et al. are not climate "skeptics". They are honest researchers studying the real effects of global warming on marine ecology. Shortly after their paper was published in January 2020, the #pruittdata story (in which a "rising star" researcher in spider ecology was caught making up tens of datasets) broke. That was when Clark and colleagues started digging into the original data from the JCU studies and finding problems. They are very pissed off about this. We all should be.
Something something hammer something something nail
Re: Fake science playing into the hands of deniers
This is really not good
Have you considered stupidity as an explanation
- Bird on a Fire
- Princess POW
- Posts: 10137
- Joined: Fri Oct 11, 2019 5:05 pm
- Location: Portugal
Re: Fake science playing into the hands of deniers
Ocean acidification is a real problem, affecting things like corals and plankton - the base of many marine food chains. It generally gets ignored, compared with the focus on temperature.
With more dissolved CO2 those organisms can't precipitate calcium carbonate and form their own skeletons, which is obviously a bit of a problem.
I'll be very annoyed if the conclusion taken from this is "ocean acidification doesn't matter" rather than "some c.nts fabricated some data about its effects on fish behaviour".
Fuckssake. How come journals get billions of research moneys every year and still haven't implemented basic automated data checking procedures?
With more dissolved CO2 those organisms can't precipitate calcium carbonate and form their own skeletons, which is obviously a bit of a problem.
I'll be very annoyed if the conclusion taken from this is "ocean acidification doesn't matter" rather than "some c.nts fabricated some data about its effects on fish behaviour".
Fuckssake. How come journals get billions of research moneys every year and still haven't implemented basic automated data checking procedures?
We have the right to a clean, healthy, sustainable environment.
Re: Fake science playing into the hands of deniers
Is it possible that the researchers were just a bit sloppy in their Excel use? Copying and pasting the previous file, inputting the new measurements, and not checking that all the old data were gone before saving as the new file. I'm not a researcher (or Excel user for that matter).
- Bird on a Fire
- Princess POW
- Posts: 10137
- Joined: Fri Oct 11, 2019 5:05 pm
- Location: Portugal
Re: Fake science playing into the hands of deniers
Well it affects lots of different files from several researchers in the same lab, so that seems unlikely, plus it goes beyond just Excel. Combined with duplicated photos that have been rotated/reversed then relabeled as a different individual, and the fact that other people present in the same labs never saw the published studies being conducted, the published details of the methods supposedly being impossible, etc etc, it does all seem a bit fishy.Pishwish wrote: ↑Sat May 08, 2021 2:32 amIs it possible that the researchers were just a bit sloppy in their Excel use? Copying and pasting the previous file, inputting the new measurements, and not checking that all the old data were gone before saving as the new file. I'm not a researcher (or Excel user for that matter).
The thing that worries me is that either these are really really stupid errors, or really really rubbish data faking. Anybody faking their data using a random number generator instead of just copying the exact same numbers dozens of times is untraceable. The huge numbers of fake images in papers is also very worrying. I expect that there's masses of research fraud being conducted all the table, mostly without ever being discovered.
We have the right to a clean, healthy, sustainable environment.
-
- Catbabel
- Posts: 738
- Joined: Sat Nov 16, 2019 8:18 am
Re: Fake science playing into the hands of deniers
I hesitate to even say the words 'Benford's Law' when sTeamie in in the thread...Bird on a Fire wrote: ↑Sat May 08, 2021 12:02 pmWell it affects lots of different files from several researchers in the same lab, so that seems unlikely, plus it goes beyond just Excel. Combined with duplicated photos that have been rotated/reversed then relabeled as a different individual, and the fact that other people present in the same labs never saw the published studies being conducted, the published details of the methods supposedly being impossible, etc etc, it does all seem a bit fishy.Pishwish wrote: ↑Sat May 08, 2021 2:32 amIs it possible that the researchers were just a bit sloppy in their Excel use? Copying and pasting the previous file, inputting the new measurements, and not checking that all the old data were gone before saving as the new file. I'm not a researcher (or Excel user for that matter).
The thing that worries me is that either these are really really stupid errors, or really really rubbish data faking. Anybody faking their data using a random number generator instead of just copying the exact same numbers dozens of times is untraceable. The huge numbers of fake images in papers is also very worrying. I expect that there's masses of research fraud being conducted all the table, mostly without ever being discovered.
- sTeamTraen
- After Pie
- Posts: 2558
- Joined: Mon Nov 11, 2019 4:24 pm
- Location: Palma de Mallorca, Spain
Re: Fake science playing into the hands of deniers
Benford's Law is in the popular imagination because of the spectacular exponential decay curve of the frequency of the digits 1-9 in the numbers that represent many natural phenomena. It can be useful for detecting financial fraud (e.g., a whole ledger of fake transactions). But there have also been some spectacular failures, e.g., election "fraud" that almost certainly wasn't (at least one general election in Iran, and also in some precincts that Biden won from Trump; in both cases, people decided that there "must" be fraud, and found it, but their analyses were based on a misunderstanding of what Benford predicts).Allo V Psycho wrote: ↑Sat May 08, 2021 1:06 pmI hesitate to even say the words 'Benford's Law' when sTeamie in in the thread...
For me the main use of Benford's Law for detecting fraud in most types of scientific data is the (roughly intuitive, but also mathematically demonstrable) observation that the second and (especially) third and subsequent digits tend to be uniformly distributed. Indeed, the first-digit part of the law only holds if the quantities in question cover several orders of magnitude, and that's only a (perhaps surprisingly) small proportion of the variables that most scientists typically want to learn about.
Something something hammer something something nail
- sTeamTraen
- After Pie
- Posts: 2558
- Joined: Mon Nov 11, 2019 4:24 pm
- Location: Palma de Mallorca, Spain
Re: Fake science playing into the hands of deniers
Ackshually, IABMCTT. It's easy to fake data with the required mean using an RNG, but if you also want a specific standard deviation, histogram, correlation matrix, and (in psychology) Cronbach alphas, you have to do quite a lot more work. Also, people often only synthesise entire fake datasets after their paper (for which they made up the summary statistics) has been published, so they have to get the SDs and other parameters right because they're right there in black and white. And they tend to be in a bit of a panic at that stage.Bird on a Fire wrote: ↑Sat May 08, 2021 12:02 pmAnybody faking their data using a random number generator instead of just copying the exact same numbers dozens of times is untraceable.
sTeamTraen wrote: ↑Sat May 08, 2021 3:35 pmBenford's Law is in the popular imagination because of the spectacular exponential decay curve of the frequency of the digits 1-9 in the numbers that represent many natural phenomena. It can be useful for detecting financial fraud (e.g., a whole ledger of fake transactions). But there have also been some spectacular failures, e.g., election "fraud" that almost certainly wasn't (at least one general election in Iran, and also in some precincts that Biden won from Trump; in both cases, people decided that there "must" be fraud, and found it, but their analyses were based on a misunderstanding of what Benford predicts).Allo V Psycho wrote: ↑Sat May 08, 2021 1:06 pmI hesitate to even say the words 'Benford's Law' when sTeamie in in the thread...
For me the main use of Benford's Law for detecting fraud in most types of scientific data is the (roughly intuitive, but also mathematically demonstrable) observation that the second and (especially) third and subsequent digits tend to be uniformly distributed. Indeed, the first-digit part of the law only holds if the quantities in question cover several orders of magnitude, and that's only a (perhaps surprisingly) small proportion of the variables that most scientists typically want to learn about.
Something something hammer something something nail
- Bird on a Fire
- Princess POW
- Posts: 10137
- Joined: Fri Oct 11, 2019 5:05 pm
- Location: Portugal
Re: Fake science playing into the hands of deniers
Maybe untraceable is an exaggeration, and for sure if you're trying to fake data to match a particular graph that's going to be tricky. (I'm not sure what's hard about specifying a SD - you have to specify all the distribution's parameters in R to get the random numbers.)
OTOH at least in my field (population biology / ecology / demography type stuff) people who develop statistical methods do loads of work on simulated data sets, often using fairly sophisticated methods to replicate both the underlying biology and subsequent sampling strategy, allowing the inclusion of known biases and observer error and stuff. You certainly couldn't pick them out by eyeballing a spreadsheet, though to be honest I've never thought to ask how easy they'd be to detect for a professional statistician.
Things like biometrics tend to be well within an order of magnitude, and I think in these papers some of the faked data was binomial/multinomial (ie, did the fish swim left or right for a bunch of replicates), which presumably isn't Benfordable?
At some point in the next few months I'll be simulating some datasets to try to replicate the timeseries I'm analysing for my PhD (not for nefarious purposes, I should add!) - would be an interesting challenge to see if people can tell the difference![Wink ;)](./images/smilies/icon_e_wink.gif)
OTOH at least in my field (population biology / ecology / demography type stuff) people who develop statistical methods do loads of work on simulated data sets, often using fairly sophisticated methods to replicate both the underlying biology and subsequent sampling strategy, allowing the inclusion of known biases and observer error and stuff. You certainly couldn't pick them out by eyeballing a spreadsheet, though to be honest I've never thought to ask how easy they'd be to detect for a professional statistician.
Things like biometrics tend to be well within an order of magnitude, and I think in these papers some of the faked data was binomial/multinomial (ie, did the fish swim left or right for a bunch of replicates), which presumably isn't Benfordable?
At some point in the next few months I'll be simulating some datasets to try to replicate the timeseries I'm analysing for my PhD (not for nefarious purposes, I should add!) - would be an interesting challenge to see if people can tell the difference
![Wink ;)](./images/smilies/icon_e_wink.gif)
We have the right to a clean, healthy, sustainable environment.
- sTeamTraen
- After Pie
- Posts: 2558
- Joined: Mon Nov 11, 2019 4:24 pm
- Location: Palma de Mallorca, Spain
Re: Fake science playing into the hands of deniers
rnorm() doesn't give you an exact match to the SD, it's only an approximation. It took quite a bit of code to get rSPRITE to give exact SDs. And then you need the correlations, and and and.Bird on a Fire wrote: ↑Sat May 08, 2021 6:01 pm(I'm not sure what's hard about specifying a SD - you have to specify all the distribution's parameters in R to get the random numbers.)
I know of a couple of R packages for simulating data. One is "faux" by the admirable Lisa de Bruine, from the admirable University of Glasgow psychology department where they teach R to all undergraduates from day 1. The other is called "wakefield".Bird on a Fire wrote: ↑Sat May 08, 2021 6:01 pmAt some point in the next few months I'll be simulating some datasets to try to replicate the timeseries I'm analysing for my PhD (not for nefarious purposes, I should add!) - would be an interesting challenge to see if people can tell the difference![]()
![Laughing :lol:](./images/smilies/icon_lol.gif)
![Laughing :lol:](./images/smilies/icon_lol.gif)
Something something hammer something something nail
- Bird on a Fire
- Princess POW
- Posts: 10137
- Joined: Fri Oct 11, 2019 5:05 pm
- Location: Portugal
Re: Fake science playing into the hands of deniers
Fair point about rnorm() - I was thinking about the ease of faking data from the get go. You could even bootstrap a published data set and add a bit of random noise to massage things in the right direction...
I should say that my group has proper audit trails for data, and it would actually be impossible for me (and I'm far too ethical anyway). But I think this shows the value of the scientific community en masse working closely with data professionals as a matter of course. IME it's the most firmly ensconced lab heads who are most resistant to this.
Not that I'm knocking the detective work done here (and elsewhere) at all! Very well done. I'd like to think spotting phoney data is generally doable, but ecology hasn't really had its Sokal moment yet and I'm unclear how well methods from other fields would translate to our kinds of data.
Now there's a fun post-thesis project for next year![Wink ;)](./images/smilies/icon_e_wink.gif)
(PS that's my new favourite R package name
)
I should say that my group has proper audit trails for data, and it would actually be impossible for me (and I'm far too ethical anyway). But I think this shows the value of the scientific community en masse working closely with data professionals as a matter of course. IME it's the most firmly ensconced lab heads who are most resistant to this.
Not that I'm knocking the detective work done here (and elsewhere) at all! Very well done. I'd like to think spotting phoney data is generally doable, but ecology hasn't really had its Sokal moment yet and I'm unclear how well methods from other fields would translate to our kinds of data.
Now there's a fun post-thesis project for next year
![Wink ;)](./images/smilies/icon_e_wink.gif)
(PS that's my new favourite R package name
![Laughing :lol:](./images/smilies/icon_lol.gif)
We have the right to a clean, healthy, sustainable environment.
Re: Fake science playing into the hands of deniers
Amusing story about reproducing data. Imagine if your exciting finding was entirely due to what colour t-shirt you were wearing that day.
-
- Catbabel
- Posts: 738
- Joined: Sat Nov 16, 2019 8:18 am
Re: Fake science playing into the hands of deniers
I have a couple! I once read a paper about the effect of having fly paper in the room on cells in culture, and I'm pretty sure that wasn't what they were trying to find - instead it came about because it finally dawned on them why their results were hard to replicate...
...and we had one when trying to isolate a biological activity which sometimes was there, and sometimes just wasn't. It was extremely ahrd work to prepare it so it was really frustrating...
Spoiler:
- shpalman
- Princess POW
- Posts: 8271
- Joined: Mon Nov 11, 2019 12:53 pm
- Location: One step beyond
- Contact:
Re: Fake science playing into the hands of deniers
Back when our lab was in Zurich I noticed our samples were worse if they were grown on Tuesdays.
having that swing is a necessary but not sufficient condition for it meaning a thing
@shpalman@mastodon.me.uk
@shpalman@mastodon.me.uk
- sTeamTraen
- After Pie
- Posts: 2558
- Joined: Mon Nov 11, 2019 4:24 pm
- Location: Palma de Mallorca, Spain
Re: Fake science playing into the hands of deniers
Not really. Part of the problem is that there's no reason to have used Excel at all. The original data must exist in some other format (the analyses and charts in the various articles clearly weren't made with Excel), and when you share data you either ask if the person who wants it can read your software's format (SPSS or whatever), or you export a CSV file and write 200 words in a cover note to explain what the variables are.Pishwish wrote: ↑Sat May 08, 2021 2:32 amIs it possible that the researchers were just a bit sloppy in their Excel use? Copying and pasting the previous file, inputting the new measurements, and not checking that all the old data were gone before saving as the new file. I'm not a researcher (or Excel user for that matter).
Excel is, however, quite good for fabricating a bunch of numbers using copy/paste. In the most egregious case, the creation date of the Excel file is 48 hours after one of the whistleblowers first wrote to the lab to ask for the data.
I suppose the defence could be that nobody in this lab with a dozen PhDs and many millions of dollars in grant money has more idea of how to use a computer for basic data analysis than my neighbour's beagle, but if they are really that incompetent then the results are immediately so untrustworthy that the articles should be retracted anyway. It reminds me of when a company is found to have done something awful and the CEO says they had no clue it was going on, but they got hired on a $20 million package because of their "unique set of competencies".
Something something hammer something something nail
-
- Snowbonk
- Posts: 551
- Joined: Wed Nov 13, 2019 12:42 pm
Re: Fake science playing into the hands of deniers
I have some experience faking 'data' to create course materials for some stats related courses I teach. Usually the best way is to decide the platonic ideal of what you want your curve or whatever to look like, then just generate the actual points around that using a normal distribution. The reported results would obviously be those arising from the fake data, not the ideal you started with, so they would perfectly match the fake data you generated. I'm amazed that these people whose careers hinge on faking data haven't figured out how to do this. It's like, 5 lines of python.sTeamTraen wrote: ↑Sat May 08, 2021 3:39 pmAckshually, IABMCTT. It's easy to fake data with the required mean using an RNG, but if you also want a specific standard deviation, histogram, correlation matrix, and (in psychology) Cronbach alphas, you have to do quite a lot more work. Also, people often only synthesise entire fake datasets after their paper (for which they made up the summary statistics) has been published, so they have to get the SDs and other parameters right because they're right there in black and white. And they tend to be in a bit of a panic at that stage.Bird on a Fire wrote: ↑Sat May 08, 2021 12:02 pmAnybody faking their data using a random number generator instead of just copying the exact same numbers dozens of times is untraceable.
- sTeamTraen
- After Pie
- Posts: 2558
- Joined: Mon Nov 11, 2019 4:24 pm
- Location: Palma de Mallorca, Spain
Re: Fake science playing into the hands of deniers
I suspect that the majority of people whose careers hinge on faking data have been drowning since undergraduate stats classes and are also utterly unable to code in any language. Most such people drop out of science fairly early on, but a few of the more sociopathic ones find ways to hang on.secret squirrel wrote: ↑Mon May 10, 2021 7:47 amI'm amazed that these people whose careers hinge on faking data haven't figured out how to do this. It's like, 5 lines of python.
Something something hammer something something nail
Re: Fake science playing into the hands of deniers
Perytons are a fun one: the Parkes radio telescope was detecting interesting transient signals with a signature that suggested they were extragalactic, and therefore extremely powerful.Allo V Psycho wrote: ↑Sun May 09, 2021 7:29 amI have a couple! I once read a paper about the effect of having fly paper in the room on cells in culture, and I'm pretty sure that wasn't what they were trying to find - instead it came about because it finally dawned on them why their results were hard to replicate...
...and we had one when trying to isolate a biological activity which sometimes was there, and sometimes just wasn't. It was extremely ahrd work to prepare it so it was really frustrating...
Spoiler:
Spoiler:
- Bird on a Fire
- Princess POW
- Posts: 10137
- Joined: Fri Oct 11, 2019 5:05 pm
- Location: Portugal
Re: Fake science playing into the hands of deniers
Exactly - use fake data then do real analyses, don't just draw some fake graphs and make up some test statistics and then have to come up with the data afterwards. Obvs.secret squirrel wrote: ↑Mon May 10, 2021 7:47 amI have some experience faking 'data' to create course materials for some stats related courses I teach. Usually the best way is to decide the platonic ideal of what you want your curve or whatever to look like, then just generate the actual points around that using a normal distribution. The reported results would obviously be those arising from the fake data, not the ideal you started with, so they would perfectly match the fake data you generated. I'm amazed that these people whose careers hinge on faking data haven't figured out how to do this. It's like, 5 lines of python.sTeamTraen wrote: ↑Sat May 08, 2021 3:39 pmAckshually, IABMCTT. It's easy to fake data with the required mean using an RNG, but if you also want a specific standard deviation, histogram, correlation matrix, and (in psychology) Cronbach alphas, you have to do quite a lot more work. Also, people often only synthesise entire fake datasets after their paper (for which they made up the summary statistics) has been published, so they have to get the SDs and other parameters right because they're right there in black and white. And they tend to be in a bit of a panic at that stage.Bird on a Fire wrote: ↑Sat May 08, 2021 12:02 pmAnybody faking their data using a random number generator instead of just copying the exact same numbers dozens of times is untraceable.
You can game Benford's Law in a single line of Python, but you have to plan first.
How would you propose distinguishing your faked data sets from the real deal?
We have the right to a clean, healthy, sustainable environment.
Re: Fake science playing into the hands of deniers
Data from my instruments tend to have a whole bunch of low level time-dependent effects in it that would distinguish it from idealized model with noise added. For example, I can see people shutting off equipment in other parts of the building, the cooling water chiller cycling on and off, and the room AC cycling on and off. These effects change depending on how many people are occupying which rooms of the lab, and the outside temperature and humidity. Normally these are all much lower level than the actual changes we are trying to measure, and so they get lumped in with the noise.
You could model them and include them in your fake data, but at that point it's probably harder than just getting the experiment to work (better*).
That's a general situation with experiments - you always draw a boundary between your experimental system and the background environment. But you can usually draw the boundary larger and try to include more background effects. One possible sign of faked data would be the lack of small non-random background signals in the timestream.
*I'm making instruments, so improving the performance is the desired outcome.
You could model them and include them in your fake data, but at that point it's probably harder than just getting the experiment to work (better*).
That's a general situation with experiments - you always draw a boundary between your experimental system and the background environment. But you can usually draw the boundary larger and try to include more background effects. One possible sign of faked data would be the lack of small non-random background signals in the timestream.
*I'm making instruments, so improving the performance is the desired outcome.
-
- After Pie
- Posts: 1621
- Joined: Mon Mar 16, 2020 4:02 am
Re: Fake science playing into the hands of deniers
There must also be people who don't set out to commit fraud, but get into it a little at a time.sTeamTraen wrote: ↑Mon May 10, 2021 9:06 amI suspect that the majority of people whose careers hinge on faking data have been drowning since undergraduate stats classes and are also utterly unable to code in any language. Most such people drop out of science fairly early on, but a few of the more sociopathic ones find ways to hang on.secret squirrel wrote: ↑Mon May 10, 2021 7:47 amI'm amazed that these people whose careers hinge on faking data haven't figured out how to do this. It's like, 5 lines of python.
- sTeamTraen
- After Pie
- Posts: 2558
- Joined: Mon Nov 11, 2019 4:24 pm
- Location: Palma de Mallorca, Spain
Re: Fake science playing into the hands of deniers
That varies from quite easy to impossible, depending on the field.Bird on a Fire wrote: ↑Mon May 10, 2021 12:25 pmHow would you propose distinguishing your faked data sets from the real deal?
In areas where you use counts of observations as the outcome variable it's almost impossible, because you typically just have to make up the four numbers in the 2x2 contingency table. If the study is of a drug trial then an investigating authority could ask for the notes of each patient, but if it's something like "When stewards at a football game wear pointy hats, the crowd are 80% more likely to comply with their directions; we used hand-held click counters to see how many people went left or right" you're basically going to need a confession. (Although I did catch someone out who claimed to have had two independent observers collecting count data this way and to have a correlation of .97 between them...)
(This is why, in the case that I mentioned in the OP, it's quite hard to prove that this didn't happen. Perhaps the best evidence is that it would simply be impossible to do the number of observations that they claimed, in the original Science paper, in less than a year of soul-destroying repetitive work, and probably more like three years. Nobody would ever actually work that way.)
But as you get into parametric statistics, it can become quite hard to fake all the aspects of a distribution. I have a case here where a psychologist changed some numbers and although he got convincing-looking means and SDs, it completely f.cked up the internal consistency (Cronbach's alpha) of the questionnaires.
Two fairly foolproof ways round that are:
1. Use an old dataset that you know (but how? bwahahahaha) is genuine and gives the right result, then just relabel the variables. This would not be hard in a "flagship" social pscyhology journal where a large number of studies are basically a 2x2 ANOVA with a significant interaction effect on a Likert-type DV. Diederik Stapel did this, and part of the evidence against him was that he had the same numbers in multiple studies that were looking at totally different phenomena. I asked him once why he didn't add and subtract the occasional 1 or 2 just to try and cover this up slightly, and he said "I'm not very good with computers".
2. Run the study, then exchange the experimental conditions of the "worst" (from your point of view) performing 5-10% of the participants in each condition. This can produce spectacular results for only a small number of flips, and all of the other parameters of the data will remain unchanged. So if you are the person charged with downloading your study's survey data from Qualtrics and you can have three minutes alone with the CSV file before anyone else in the team sees it, you can have whatever results you want.
Something something hammer something something nail
- sTeamTraen
- After Pie
- Posts: 2558
- Joined: Mon Nov 11, 2019 4:24 pm
- Location: Palma de Mallorca, Spain
Re: Fake science playing into the hands of deniers
Yes. Diederik Stapel describes this quite well in his book [PDF]. Once you discover that you can get any result you want, you have less and less incentive to do the work honestly; plus, you start to become known for your high productivity and great results, and if you stopped faking you can imagine that people might ask where your magic touch went to.Millennie Al wrote: ↑Tue May 11, 2021 12:35 amThere must also be people who don't set out to commit fraud, but get into it a little at a time.sTeamTraen wrote: ↑Mon May 10, 2021 9:06 amI suspect that the majority of people whose careers hinge on faking data have been drowning since undergraduate stats classes and are also utterly unable to code in any language. Most such people drop out of science fairly early on, but a few of the more sociopathic ones find ways to hang on.secret squirrel wrote: ↑Mon May 10, 2021 7:47 amI'm amazed that these people whose careers hinge on faking data haven't figured out how to do this. It's like, 5 lines of python.
Something something hammer something something nail
- shpalman
- Princess POW
- Posts: 8271
- Joined: Mon Nov 11, 2019 12:53 pm
- Location: One step beyond
- Contact:
Re: Fake science playing into the hands of deniers
See also the Report of the Investigation Committee on the possibility of Scientific Misconduct in the work of Hendrik Schön and Coauthors by Malcolm R. Beasley et al.
Basically starts by reusing the same data in different figures and ends up completely simulating the "data".
Basically starts by reusing the same data in different figures and ends up completely simulating the "data".
having that swing is a necessary but not sufficient condition for it meaning a thing
@shpalman@mastodon.me.uk
@shpalman@mastodon.me.uk
Re: Fake science playing into the hands of deniers
I'm guessing that a gateway is being asked for the data for something by a student, and not being able to find it in your mess of a file system*, and realizing that you can just generate something they can play with instead.
*Paper or electronic.
*Paper or electronic.