Page 1 of 1
Papers without code
Posted: Mon Mar 08, 2021 12:26 pm
by Stephanie
Might be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce
https://thenextweb.com/neural/2021/03/0 ... ndication/
Re: Papers without code
Posted: Tue Mar 09, 2021 1:34 pm
by kerrya1
Yet another example of the Reproducibility Crisis in research, and the reason I'll have a job until retirement!
Re: Papers without code
Posted: Thu Mar 11, 2021 9:50 am
by jimbob
kerrya1 wrote: ↑Tue Mar 09, 2021 1:34 pm
Yet another example of the Reproducibility Crisis in research, and the reason I'll have a job until retirement!
See also some circuits developed with evolutionary algorithms. They can be very effective, but are resistant to analysis as they often utilise strange parasitic effects that simulations don't account for. They might be efficient both in silicon utilisation and power, but it's very hard to assess corner situations for reliability assesment.
Re: Papers without code
Posted: Thu Mar 11, 2021 12:46 pm
by tom p
That is very interesting, thanks Steph.
we're working on a project that should result in a paper.
I'll make sure we publish everything necessary for our reuslts to be reproduced
Re: Papers without code
Posted: Thu Mar 11, 2021 4:54 pm
by sTeamTraen
Non-reproducibility has been a big problem in computer science for quite some time; even if code is provided it doesn't always work.
For
this paper I had to get a model to run that contained about 20,000 lines of Python 2.7 code (!) and it was a major ordeal. First, the code wasn't provided where the authors had said they'd put it, although I eventually found it by googling. Then I had to build a Linux VM to run it in, which I was able to work out how to do but which may not be within reach of every psychologist. Then the models took 36 hours each to run, and there were four of them. And the numbers they output were similar, but not identical, to what was reported in the paper. So the original article was to some extent unfalsifiable in the sense that without help from the authors nobody could get the same results (and the code is a bit f.cked now because it seems that Python 2.x is going the way of all flesh), but there was so much else wrong with the whole idea that that didn't matter too much.
Re: Papers without code
Posted: Thu Mar 11, 2021 5:57 pm
by shpalman
My preferred way to run Python on Windows is to use Ubuntu in the WSL.
But anyone stuck on python 2.x who isn't able to update their code to run on python 3 shouldn't be allowed near a computer just in general.
They probably did division of integers at some point in it or something. And I remember that thing where windows and linux present the files in different orders via their OS calls or something, which ruined a whole load of people's sh.t.
Re: Papers without code
Posted: Fri Mar 12, 2021 12:21 am
by dyqik
If the code doesn't have unit tests to detect that when run under a multi-environnent testing system, then it's not reliable enough for publication.
Re: Papers without code
Posted: Fri Mar 12, 2021 6:11 pm
by Stephanie
tom p wrote: ↑Thu Mar 11, 2021 12:46 pm
That is very interesting, thanks Steph.
we're working on a project that should result in a paper.
I'll make sure we publish everything necessary for our reuslts to be reproduced
oh that sounds cool :) you'll have to share with us, if you can