Papers without code
- Stephanie
- Stummy Beige
- Posts: 2902
- Joined: Wed Sep 25, 2019 4:38 pm
- Location: clinging tenaciously to your buttocks
Papers without code
Might be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce
https://thenextweb.com/neural/2021/03/0 ... ndication/
https://thenextweb.com/neural/2021/03/0 ... ndication/
"I got a flu virus named after me 'cause I kissed a bat on a dare."
Re: Papers without code
Yet another example of the Reproducibility Crisis in research, and the reason I'll have a job until retirement!Stephanie wrote: ↑Mon Mar 08, 2021 12:26 pmMight be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce
https://thenextweb.com/neural/2021/03/0 ... ndication/
Re: Papers without code
See also some circuits developed with evolutionary algorithms. They can be very effective, but are resistant to analysis as they often utilise strange parasitic effects that simulations don't account for. They might be efficient both in silicon utilisation and power, but it's very hard to assess corner situations for reliability assesment.kerrya1 wrote: ↑Tue Mar 09, 2021 1:34 pmYet another example of the Reproducibility Crisis in research, and the reason I'll have a job until retirement!Stephanie wrote: ↑Mon Mar 08, 2021 12:26 pmMight be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce
https://thenextweb.com/neural/2021/03/0 ... ndication/
Have you considered stupidity as an explanation
Re: Papers without code
That is very interesting, thanks Steph.Stephanie wrote: ↑Mon Mar 08, 2021 12:26 pmMight be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce
https://thenextweb.com/neural/2021/03/0 ... ndication/
we're working on a project that should result in a paper.
I'll make sure we publish everything necessary for our reuslts to be reproduced
- sTeamTraen
- After Pie
- Posts: 2558
- Joined: Mon Nov 11, 2019 4:24 pm
- Location: Palma de Mallorca, Spain
Re: Papers without code
Non-reproducibility has been a big problem in computer science for quite some time; even if code is provided it doesn't always work.
For this paper I had to get a model to run that contained about 20,000 lines of Python 2.7 code (!) and it was a major ordeal. First, the code wasn't provided where the authors had said they'd put it, although I eventually found it by googling. Then I had to build a Linux VM to run it in, which I was able to work out how to do but which may not be within reach of every psychologist. Then the models took 36 hours each to run, and there were four of them. And the numbers they output were similar, but not identical, to what was reported in the paper. So the original article was to some extent unfalsifiable in the sense that without help from the authors nobody could get the same results (and the code is a bit f.cked now because it seems that Python 2.x is going the way of all flesh), but there was so much else wrong with the whole idea that that didn't matter too much.
For this paper I had to get a model to run that contained about 20,000 lines of Python 2.7 code (!) and it was a major ordeal. First, the code wasn't provided where the authors had said they'd put it, although I eventually found it by googling. Then I had to build a Linux VM to run it in, which I was able to work out how to do but which may not be within reach of every psychologist. Then the models took 36 hours each to run, and there were four of them. And the numbers they output were similar, but not identical, to what was reported in the paper. So the original article was to some extent unfalsifiable in the sense that without help from the authors nobody could get the same results (and the code is a bit f.cked now because it seems that Python 2.x is going the way of all flesh), but there was so much else wrong with the whole idea that that didn't matter too much.
Something something hammer something something nail
- shpalman
- Princess POW
- Posts: 8271
- Joined: Mon Nov 11, 2019 12:53 pm
- Location: One step beyond
- Contact:
Re: Papers without code
My preferred way to run Python on Windows is to use Ubuntu in the WSL.
But anyone stuck on python 2.x who isn't able to update their code to run on python 3 shouldn't be allowed near a computer just in general.
They probably did division of integers at some point in it or something. And I remember that thing where windows and linux present the files in different orders via their OS calls or something, which ruined a whole load of people's sh.t.
But anyone stuck on python 2.x who isn't able to update their code to run on python 3 shouldn't be allowed near a computer just in general.
They probably did division of integers at some point in it or something. And I remember that thing where windows and linux present the files in different orders via their OS calls or something, which ruined a whole load of people's sh.t.
having that swing is a necessary but not sufficient condition for it meaning a thing
@shpalman@mastodon.me.uk
@shpalman@mastodon.me.uk
Re: Papers without code
If the code doesn't have unit tests to detect that when run under a multi-environnent testing system, then it's not reliable enough for publication.
- Stephanie
- Stummy Beige
- Posts: 2902
- Joined: Wed Sep 25, 2019 4:38 pm
- Location: clinging tenaciously to your buttocks
Re: Papers without code
oh that sounds cool :) you'll have to share with us, if you cantom p wrote: ↑Thu Mar 11, 2021 12:46 pmThat is very interesting, thanks Steph.Stephanie wrote: ↑Mon Mar 08, 2021 12:26 pmMight be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce
https://thenextweb.com/neural/2021/03/0 ... ndication/
we're working on a project that should result in a paper.
I'll make sure we publish everything necessary for our reuslts to be reproduced
"I got a flu virus named after me 'cause I kissed a bat on a dare."