Papers without code

Get your science fix here: research, quackery, activism and all the rest
Post Reply
User avatar
Stephanie
Stummy Beige
Posts: 2896
Joined: Wed Sep 25, 2019 4:38 pm
Location: clinging tenaciously to your buttocks

Papers without code

Post by Stephanie » Mon Mar 08, 2021 12:26 pm

Might be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce

https://thenextweb.com/neural/2021/03/0 ... ndication/
"I got a flu virus named after me 'cause I kissed a bat on a dare."

kerrya1
Clardic Fug
Posts: 184
Joined: Tue Nov 12, 2019 11:13 am

Re: Papers without code

Post by kerrya1 » Tue Mar 09, 2021 1:34 pm

Stephanie wrote:
Mon Mar 08, 2021 12:26 pm
Might be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce

https://thenextweb.com/neural/2021/03/0 ... ndication/
Yet another example of the Reproducibility Crisis in research, and the reason I'll have a job until retirement!

User avatar
jimbob
Light of Blast
Posts: 5276
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: Papers without code

Post by jimbob » Thu Mar 11, 2021 9:50 am

kerrya1 wrote:
Tue Mar 09, 2021 1:34 pm
Stephanie wrote:
Mon Mar 08, 2021 12:26 pm
Might be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce

https://thenextweb.com/neural/2021/03/0 ... ndication/
Yet another example of the Reproducibility Crisis in research, and the reason I'll have a job until retirement!
See also some circuits developed with evolutionary algorithms. They can be very effective, but are resistant to analysis as they often utilise strange parasitic effects that simulations don't account for. They might be efficient both in silicon utilisation and power, but it's very hard to assess corner situations for reliability assesment.
Have you considered stupidity as an explanation

tom p
After Pie
Posts: 1876
Joined: Mon Nov 11, 2019 1:14 pm
Location: the low countries

Re: Papers without code

Post by tom p » Thu Mar 11, 2021 12:46 pm

Stephanie wrote:
Mon Mar 08, 2021 12:26 pm
Might be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce

https://thenextweb.com/neural/2021/03/0 ... ndication/
That is very interesting, thanks Steph.
we're working on a project that should result in a paper.
I'll make sure we publish everything necessary for our reuslts to be reproduced

User avatar
sTeamTraen
After Pie
Posts: 2550
Joined: Mon Nov 11, 2019 4:24 pm
Location: Palma de Mallorca, Spain

Re: Papers without code

Post by sTeamTraen » Thu Mar 11, 2021 4:54 pm

Non-reproducibility has been a big problem in computer science for quite some time; even if code is provided it doesn't always work.

For this paper I had to get a model to run that contained about 20,000 lines of Python 2.7 code (!) and it was a major ordeal. First, the code wasn't provided where the authors had said they'd put it, although I eventually found it by googling. Then I had to build a Linux VM to run it in, which I was able to work out how to do but which may not be within reach of every psychologist. Then the models took 36 hours each to run, and there were four of them. And the numbers they output were similar, but not identical, to what was reported in the paper. So the original article was to some extent unfalsifiable in the sense that without help from the authors nobody could get the same results (and the code is a bit f.cked now because it seems that Python 2.x is going the way of all flesh), but there was so much else wrong with the whole idea that that didn't matter too much.
Something something hammer something something nail

User avatar
shpalman
Princess POW
Posts: 8241
Joined: Mon Nov 11, 2019 12:53 pm
Location: One step beyond
Contact:

Re: Papers without code

Post by shpalman » Thu Mar 11, 2021 5:57 pm

My preferred way to run Python on Windows is to use Ubuntu in the WSL.

But anyone stuck on python 2.x who isn't able to update their code to run on python 3 shouldn't be allowed near a computer just in general.

They probably did division of integers at some point in it or something. And I remember that thing where windows and linux present the files in different orders via their OS calls or something, which ruined a whole load of people's sh.t.
having that swing is a necessary but not sufficient condition for it meaning a thing
@shpalman@mastodon.me.uk

User avatar
dyqik
Princess POW
Posts: 7524
Joined: Wed Sep 25, 2019 4:19 pm
Location: Masshole
Contact:

Re: Papers without code

Post by dyqik » Fri Mar 12, 2021 12:21 am

If the code doesn't have unit tests to detect that when run under a multi-environnent testing system, then it's not reliable enough for publication.

User avatar
Stephanie
Stummy Beige
Posts: 2896
Joined: Wed Sep 25, 2019 4:38 pm
Location: clinging tenaciously to your buttocks

Re: Papers without code

Post by Stephanie » Fri Mar 12, 2021 6:11 pm

tom p wrote:
Thu Mar 11, 2021 12:46 pm
Stephanie wrote:
Mon Mar 08, 2021 12:26 pm
Might be of interest to some folk here, about problems in the machine learning community with papers that don't have the necessary code or methods for other researchers to reproduce

https://thenextweb.com/neural/2021/03/0 ... ndication/
That is very interesting, thanks Steph.
we're working on a project that should result in a paper.
I'll make sure we publish everything necessary for our reuslts to be reproduced
oh that sounds cool :) you'll have to share with us, if you can
"I got a flu virus named after me 'cause I kissed a bat on a dare."

Post Reply