There is too much science for science to work properly

Get your science fix here: research, quackery, activism and all the rest
Post Reply
User avatar
Bird on a Fire
Princess POW
Posts: 10137
Joined: Fri Oct 11, 2019 5:05 pm
Location: Portugal

There is too much science for science to work properly

Post by Bird on a Fire » Fri Oct 08, 2021 7:06 pm

According to this paper in PNAS:
In many academic fields, the number of papers published each year has increased significantly over time. Policy measures aim to increase the quantity of scientists, research funding, and scientific output, which is measured by the number of papers produced. These quantitative metrics determine the career trajectories of scholars and evaluations of academic departments, institutions, and nations. Whether and how these increases in the numbers of scientists and papers translate into advances in knowledge is unclear, however. Here, we first lay out a theoretical argument for why too many papers published each year in a field can lead to stagnation rather than advance. The deluge of new papers may deprive reviewers and readers the cognitive slack required to fully recognize and understand novel ideas. Competition among many new ideas may prevent the gradual accumulation of focused attention on a promising new idea. Then, we show data supporting the predictions of this theory. When the number of papers published per year in a scientific field grows large, citations flow disproportionately to already well-cited papers; the list of most-cited papers ossifies; new papers are unlikely to ever become highly cited, and when they do, it is not through a gradual, cumulative process of attention gathering; and newly published papers become unlikely to disrupt existing work. These findings suggest that the progress of large scientific fields may be slowed, trapped in existing canon. Policy measures shifting how scientific work is produced, disseminated, consumed, and rewarded may be called for to push fields into new, more fertile areas of study.
Perhaps turning researchers into content providers for large publishing corporations wasn't the best economic model to incentivise scientific progress after all.
We have the right to a clean, healthy, sustainable environment.

User avatar
Fishnut
After Pie
Posts: 2447
Joined: Mon Nov 11, 2019 1:15 pm
Location: UK

Re: There is too much science for science to work properly

Post by Fishnut » Fri Oct 08, 2021 7:42 pm

That sounds like a really interesting paper.

I remember reading this Guardian Long Read a few years ago (I can't remember if it was discussed in previous iterations of this forum or not) and being blown away. I had no idea that Rupert f.cking Murdoch was responsible for the proliferation of journals, but it does make the insane model of researchers do to the work, researchers peer-review others' work for free, researchers pay to publish their work and researchers pay to access the work of others while the publishers make a fortune make much more sense.
it's okay to say "I don't know"

jeremy
Ghastly Pink
Posts: 4
Joined: Sun Nov 17, 2019 4:15 pm

Re: There is too much science for science to work properly

Post by jeremy » Sat Oct 09, 2021 2:56 pm

Fishnut wrote:
Fri Oct 08, 2021 7:42 pm
That sounds like a really interesting paper.

I remember reading this Guardian Long Read a few years ago (I can't remember if it was discussed in previous iterations of this forum or not) and being blown away. I had no idea that Rupert f.cking Murdoch was responsible for the proliferation of journals, but it does make the insane model of researchers do to the work, researchers peer-review others' work for free, researchers pay to publish their work and researchers pay to access the work of others while the publishers make a fortune make much more sense.
I think you mean Robert f.cking Maxwell.

User avatar
Fishnut
After Pie
Posts: 2447
Joined: Mon Nov 11, 2019 1:15 pm
Location: UK

Re: There is too much science for science to work properly

Post by Fishnut » Sat Oct 09, 2021 9:46 pm

jeremy wrote:
Sat Oct 09, 2021 2:56 pm
Fishnut wrote:
Fri Oct 08, 2021 7:42 pm
That sounds like a really interesting paper.

I remember reading this Guardian Long Read a few years ago (I can't remember if it was discussed in previous iterations of this forum or not) and being blown away. I had no idea that Rupert f.cking Murdoch was responsible for the proliferation of journals, but it does make the insane model of researchers do to the work, researchers peer-review others' work for free, researchers pay to publish their work and researchers pay to access the work of others while the publishers make a fortune make much more sense.
I think you mean Robert f.cking Maxwell.
You're right, I did. I even checked the article to make sure I had the right f.cker and still managed to mix them up.
it's okay to say "I don't know"

User avatar
Bird on a Fire
Princess POW
Posts: 10137
Joined: Fri Oct 11, 2019 5:05 pm
Location: Portugal

Re: There is too much science for science to work properly

Post by Bird on a Fire » Sat Oct 09, 2021 10:22 pm

A couple of times in my PhD I've tried to stay on top of the literature by getting table of contents and alerts for all the journals that regularly publish my kind of stuff.

It was hours a week just to sift through titles, ending with a million tabs for abstracts I'd skim, and end up wanting to read about 10% of them, of which I could actually access less than half.

These days I just rely on tweets, emails and chatting to people I know. And I think everyone else does too.
We have the right to a clean, healthy, sustainable environment.

User avatar
sTeamTraen
After Pie
Posts: 2551
Joined: Mon Nov 11, 2019 4:24 pm
Location: Palma de Mallorca, Spain

Re: There is too much science for science to work properly

Post by sTeamTraen » Sat Oct 09, 2021 10:35 pm

One of my hobbies at conferences is suggesting to my fellow psychologists that we would all be better off with 90% fewer researchers all getting 10x as much grant money so we could run decent sample sizes. Individually they are surprisingly receptive to this idea, presumably because they all imagine that other people would make up the 90%.
Something something hammer something something nail

User avatar
dyqik
Princess POW
Posts: 7526
Joined: Wed Sep 25, 2019 4:19 pm
Location: Masshole
Contact:

Re: There is too much science for science to work properly

Post by dyqik » Sun Oct 10, 2021 12:27 am

I can't remember that last time I read a journal or astro-ph.

User avatar
Woodchopper
Princess POW
Posts: 7057
Joined: Sat Oct 12, 2019 9:05 am

Re: There is too much science for science to work properly

Post by Woodchopper » Sun Oct 10, 2021 11:15 am

Bird on a Fire wrote:
Sat Oct 09, 2021 10:22 pm
A couple of times in my PhD I've tried to stay on top of the literature by getting table of contents and alerts for all the journals that regularly publish my kind of stuff.

It was hours a week just to sift through titles, ending with a million tabs for abstracts I'd skim, and end up wanting to read about 10% of them, of which I could actually access less than half.

These days I just rely on tweets, emails and chatting to people I know. And I think everyone else does too.
Once you get your PhD it'll get a lot easier. Just set up a Google Scholar alert for anything that cites one of your publications. Ignore the rest as irrelevant.

User avatar
Stephanie
Stummy Beige
Posts: 2896
Joined: Wed Sep 25, 2019 4:38 pm
Location: clinging tenaciously to your buttocks

Re: There is too much science for science to work properly

Post by Stephanie » Sun Oct 10, 2021 3:04 pm

I remember reading a commentary about a psychological theory that included this. I think about it quite a lot.
As a doctoral student, I read all of psychology whether it be biological, social, developmental or cognitive. I read philosophy, sociology and psychiatry and I even read novels related in all sorts of ways to the human condition. Now my students drown in the literature. We have published so many studies, so many theories in so many journals and books that they cannot possibly grasp it all and an academic learned helplessness develops. So the solution is known as ‘being focused’ or ‘specialising’ and similarly academic careers are based on ‘having a reputation in a field’ or making a contribution ‘to an area’. Yet I have examined students whose subject is cancer who have never read the literature on heart disease, students who study smoking but have ‘focused only’ on the qualitative research and those who only cite research from the UK as this is where their study is. And likewise many researchers may know their ‘field’ and are known ‘for an area’ but remain blissfully unaware of all the other literature that could enrich their work. And then these areas become inbred, stunted and stale as there is no chance of cross-fertilisation as they know no other area to cross-fertilise with. The vastness of the literature therefore means that it remains unread causing a narrowing vision of what psychology is and our research becomes increasingly obvious.
"I got a flu virus named after me 'cause I kissed a bat on a dare."

User avatar
Woodchopper
Princess POW
Posts: 7057
Joined: Sat Oct 12, 2019 9:05 am

Re: There is too much science for science to work properly

Post by Woodchopper » Wed Oct 13, 2021 6:28 am

Stephanie wrote:
Sun Oct 10, 2021 3:04 pm
I remember reading a commentary about a psychological theory that included this. I think about it quite a lot.
As a doctoral student, I read all of psychology whether it be biological, social, developmental or cognitive. I read philosophy, sociology and psychiatry and I even read novels related in all sorts of ways to the human condition. Now my students drown in the literature. We have published so many studies, so many theories in so many journals and books that they cannot possibly grasp it all and an academic learned helplessness develops. So the solution is known as ‘being focused’ or ‘specialising’ and similarly academic careers are based on ‘having a reputation in a field’ or making a contribution ‘to an area’. Yet I have examined students whose subject is cancer who have never read the literature on heart disease, students who study smoking but have ‘focused only’ on the qualitative research and those who only cite research from the UK as this is where their study is. And likewise many researchers may know their ‘field’ and are known ‘for an area’ but remain blissfully unaware of all the other literature that could enrich their work. And then these areas become inbred, stunted and stale as there is no chance of cross-fertilisation as they know no other area to cross-fertilise with. The vastness of the literature therefore means that it remains unread causing a narrowing vision of what psychology is and our research becomes increasingly obvious.
Well yes and no.

I agree that so much is published that would take a huge amount of time for someone to master a subject beyond the field in which they work, and those fields tend to be defined pretty narrowly.

But its not difficult to use someone else's mastery. Scientists regularly publish summaries of the literature in their field, and so systematic reviews, meta analysis etc have a vital role to play in research. As I advised a PhD student a couple of days ago, its not necessary to read a huge number of articles from another field, just go to Google Scholar and find someone else's literature review. Its also possible to get a basic understanding of the key findings in a field from reading textbooks or publications designed for a lay audience (I'm thinking of reports written by academics more than popular science bestsellers). Paywalls used to be a problem, but vast amounts of material are now available for free or for very little money.

Lots of methods are also sufficiently similar that people can appraise a paper and understand some of its basic strengths and weaknesses even if they are not experts in that field - for example similar statistical methods are used in fields as diverse as economics, medicine and ecology. So an expert on cancer should be able to read a paper on heart disease (though of course that doesn't apply elsewhere, a biologist might struggle to understand a physics paper).

Assuming that the students mentioned are doing graduate degrees I think that the problem is more a culture of intellectual Fordism in which students are treated as items on a production line. In that environment reading widely can be seen as a distraction from the purpose of getting their degree certificate and necessary number of publications on time.

User avatar
dyqik
Princess POW
Posts: 7526
Joined: Wed Sep 25, 2019 4:19 pm
Location: Masshole
Contact:

Re: There is too much science for science to work properly

Post by dyqik » Wed Oct 13, 2021 3:51 pm

Stephanie wrote:
Sun Oct 10, 2021 3:04 pm
I remember reading a commentary about a psychological theory that included this. I think about it quite a lot.
As a doctoral student, I read all of psychology whether it be biological, social, developmental or cognitive. I read philosophy, sociology and psychiatry and I even read novels related in all sorts of ways to the human condition. Now my students drown in the literature. We have published so many studies, so many theories in so many journals and books that they cannot possibly grasp it all and an academic learned helplessness develops. So the solution is known as ‘being focused’ or ‘specialising’ and similarly academic careers are based on ‘having a reputation in a field’ or making a contribution ‘to an area’. Yet I have examined students whose subject is cancer who have never read the literature on heart disease, students who study smoking but have ‘focused only’ on the qualitative research and those who only cite research from the UK as this is where their study is. And likewise many researchers may know their ‘field’ and are known ‘for an area’ but remain blissfully unaware of all the other literature that could enrich their work. And then these areas become inbred, stunted and stale as there is no chance of cross-fertilisation as they know no other area to cross-fertilise with. The vastness of the literature therefore means that it remains unread causing a narrowing vision of what psychology is and our research becomes increasingly obvious.
This is why review articles and graduate level textbooks exist.

User avatar
Woodchopper
Princess POW
Posts: 7057
Joined: Sat Oct 12, 2019 9:05 am

Re: There is too much science for science to work properly

Post by Woodchopper » Wed Oct 13, 2021 3:53 pm

dyqik wrote:
Wed Oct 13, 2021 3:51 pm
Stephanie wrote:
Sun Oct 10, 2021 3:04 pm
I remember reading a commentary about a psychological theory that included this. I think about it quite a lot.
As a doctoral student, I read all of psychology whether it be biological, social, developmental or cognitive. I read philosophy, sociology and psychiatry and I even read novels related in all sorts of ways to the human condition. Now my students drown in the literature. We have published so many studies, so many theories in so many journals and books that they cannot possibly grasp it all and an academic learned helplessness develops. So the solution is known as ‘being focused’ or ‘specialising’ and similarly academic careers are based on ‘having a reputation in a field’ or making a contribution ‘to an area’. Yet I have examined students whose subject is cancer who have never read the literature on heart disease, students who study smoking but have ‘focused only’ on the qualitative research and those who only cite research from the UK as this is where their study is. And likewise many researchers may know their ‘field’ and are known ‘for an area’ but remain blissfully unaware of all the other literature that could enrich their work. And then these areas become inbred, stunted and stale as there is no chance of cross-fertilisation as they know no other area to cross-fertilise with. The vastness of the literature therefore means that it remains unread causing a narrowing vision of what psychology is and our research becomes increasingly obvious.
This is why review articles and graduate level textbooks exist.
Yes, you just summed up my 300 word post in a sentence.

User avatar
Bird on a Fire
Princess POW
Posts: 10137
Joined: Fri Oct 11, 2019 5:05 pm
Location: Portugal

Re: There is too much science for science to work properly

Post by Bird on a Fire » Wed Oct 13, 2021 9:02 pm

dyqik wrote:
Wed Oct 13, 2021 3:51 pm
Stephanie wrote:
Sun Oct 10, 2021 3:04 pm
I remember reading a commentary about a psychological theory that included this. I think about it quite a lot.
As a doctoral student, I read all of psychology whether it be biological, social, developmental or cognitive. I read philosophy, sociology and psychiatry and I even read novels related in all sorts of ways to the human condition. Now my students drown in the literature. We have published so many studies, so many theories in so many journals and books that they cannot possibly grasp it all and an academic learned helplessness develops. So the solution is known as ‘being focused’ or ‘specialising’ and similarly academic careers are based on ‘having a reputation in a field’ or making a contribution ‘to an area’. Yet I have examined students whose subject is cancer who have never read the literature on heart disease, students who study smoking but have ‘focused only’ on the qualitative research and those who only cite research from the UK as this is where their study is. And likewise many researchers may know their ‘field’ and are known ‘for an area’ but remain blissfully unaware of all the other literature that could enrich their work. And then these areas become inbred, stunted and stale as there is no chance of cross-fertilisation as they know no other area to cross-fertilise with. The vastness of the literature therefore means that it remains unread causing a narrowing vision of what psychology is and our research becomes increasingly obvious.
This is why review articles and graduate level textbooks exist.
Well sort of, but do many students (or other researchers) actually tend to read reviews and textbooks from outside their immediate field?
We have the right to a clean, healthy, sustainable environment.

User avatar
dyqik
Princess POW
Posts: 7526
Joined: Wed Sep 25, 2019 4:19 pm
Location: Masshole
Contact:

Re: There is too much science for science to work properly

Post by dyqik » Thu Oct 14, 2021 12:12 am

No, but if you decided that you wanted to know something about something outside your field, it's where you would start.

User avatar
Lariliss
Navel Tan
Posts: 5
Joined: Fri Oct 29, 2021 3:24 pm

Re: There is too much science for science to work properly

Post by Lariliss » Wed Nov 03, 2021 1:10 pm

Bird on a Fire wrote:
Fri Oct 08, 2021 7:06 pm
According to this paper in PNAS:
In many academic fields, the number of papers published each year has increased significantly over time. Policy measures aim to increase the quantity of scientists, research funding, and scientific output, which is measured by the number of papers produced. These quantitative metrics determine the career trajectories of scholars and evaluations of academic departments, institutions, and nations. Whether and how these increases in the numbers of scientists and papers translate into advances in knowledge is unclear, however. Here, we first lay out a theoretical argument for why too many papers published each year in a field can lead to stagnation rather than advance. The deluge of new papers may deprive reviewers and readers the cognitive slack required to fully recognize and understand novel ideas. Competition among many new ideas may prevent the gradual accumulation of focused attention on a promising new idea. Then, we show data supporting the predictions of this theory. When the number of papers published per year in a scientific field grows large, citations flow disproportionately to already well-cited papers; the list of most-cited papers ossifies; new papers are unlikely to ever become highly cited, and when they do, it is not through a gradual, cumulative process of attention gathering; and newly published papers become unlikely to disrupt existing work. These findings suggest that the progress of large scientific fields may be slowed, trapped in existing canon. Policy measures shifting how scientific work is produced, disseminated, consumed, and rewarded may be called for to push fields into new, more fertile areas of study.
Perhaps turning researchers into content providers for large publishing corporations wasn't the best economic model to incentivise scientific progress after all.
'Content provider’ might not be an exact term. Since you are on a scientific path you should publish papers regularly (is it a trap or trade-off for annual score check).
The main question is for the publications to be truthful (it was checked after the 2017 issue raised of p-hacking).
Nevertheless, frequency of publications is an indicator of the certain research activity, actuality, expediency and uniqueness.
Today’s science leaps (and technology projects, which often depend on them) need fast reaction, awareness and cooperation.
Number, Letter, Note: Know, Think, Dream.

Allo V Psycho
Catbabel
Posts: 734
Joined: Sat Nov 16, 2019 8:18 am

Re: There is too much science for science to work properly

Post by Allo V Psycho » Sun Nov 07, 2021 4:45 pm

Bird on a Fire wrote:
Fri Oct 08, 2021 7:06 pm
According to this paper in PNAS:
In many academic fields, the number of papers published each year has increased significantly over time. Policy measures aim to increase the quantity of scientists, research funding, and scientific output, which is measured by the number of papers produced. These quantitative metrics determine the career trajectories of scholars and evaluations of academic departments, institutions, and nations. Whether and how these increases in the numbers of scientists and papers translate into advances in knowledge is unclear, however. Here, we first lay out a theoretical argument for why too many papers published each year in a field can lead to stagnation rather than advance. The deluge of new papers may deprive reviewers and readers the cognitive slack required to fully recognize and understand novel ideas. Competition among many new ideas may prevent the gradual accumulation of focused attention on a promising new idea. Then, we show data supporting the predictions of this theory. When the number of papers published per year in a scientific field grows large, citations flow disproportionately to already well-cited papers; the list of most-cited papers ossifies; new papers are unlikely to ever become highly cited, and when they do, it is not through a gradual, cumulative process of attention gathering; and newly published papers become unlikely to disrupt existing work. These findings suggest that the progress of large scientific fields may be slowed, trapped in existing canon. Policy measures shifting how scientific work is produced, disseminated, consumed, and rewarded may be called for to push fields into new, more fertile areas of study.
Perhaps turning researchers into content providers for large publishing corporations wasn't the best economic model to incentivise scientific progress after all.
Just been e-chatting to a friend who is about as eminent in British science as you can get, and he sounds very disenchanted with his field: not just the volume of papers, but the relative triviality of so many of them. Often they are using extremely sophisticated techniques "a gold-plated sledge hammer for a very small nut" just because they can, even if it could be done more simply. "All this stuff", he called it, emphasis in the original, and "if you want to maintain a broad interest the minutiae are impossible to remember".

User avatar
Stephanie
Stummy Beige
Posts: 2896
Joined: Wed Sep 25, 2019 4:38 pm
Location: clinging tenaciously to your buttocks

Re: There is too much science for science to work properly

Post by Stephanie » Mon Nov 08, 2021 5:14 pm

dyqik wrote:
Wed Oct 13, 2021 3:51 pm
Stephanie wrote:
Sun Oct 10, 2021 3:04 pm
I remember reading a commentary about a psychological theory that included this. I think about it quite a lot.
As a doctoral student, I read all of psychology whether it be biological, social, developmental or cognitive. I read philosophy, sociology and psychiatry and I even read novels related in all sorts of ways to the human condition. Now my students drown in the literature. We have published so many studies, so many theories in so many journals and books that they cannot possibly grasp it all and an academic learned helplessness develops. So the solution is known as ‘being focused’ or ‘specialising’ and similarly academic careers are based on ‘having a reputation in a field’ or making a contribution ‘to an area’. Yet I have examined students whose subject is cancer who have never read the literature on heart disease, students who study smoking but have ‘focused only’ on the qualitative research and those who only cite research from the UK as this is where their study is. And likewise many researchers may know their ‘field’ and are known ‘for an area’ but remain blissfully unaware of all the other literature that could enrich their work. And then these areas become inbred, stunted and stale as there is no chance of cross-fertilisation as they know no other area to cross-fertilise with. The vastness of the literature therefore means that it remains unread causing a narrowing vision of what psychology is and our research becomes increasingly obvious.
This is why review articles and graduate level textbooks exist.
I probably should have clarified - this is from health psychology - so heart disease, smoking, etc, would all be relevant
"I got a flu virus named after me 'cause I kissed a bat on a dare."

Lydia Gwilt
Stargoon
Posts: 140
Joined: Mon Nov 11, 2019 2:05 pm

Re: There is too much science for science to work properly

Post by Lydia Gwilt » Fri Nov 12, 2021 9:23 am

Woodchopper wrote:
Wed Oct 13, 2021 6:28 am

But its not difficult to use someone else's mastery. Scientists regularly publish summaries of the literature in their field, and so systematic reviews, meta analysis etc have a vital role to play in research. As I advised a PhD student a couple of days ago, its not necessary to read a huge number of articles from another field, just go to Google Scholar and find someone else's literature review. Its also possible to get a basic understanding of the key findings in a field from reading textbooks or publications designed for a lay audience (I'm thinking of reports written by academics more than popular science bestsellers). Paywalls used to be a problem, but vast amounts of material are now available for free or for very little money.
I agree that reviews are a great place to start, but there is always a risk of things spiralling in on themselves, with people writing reviews based on reading only other reviews. That is a good way of consolidating one interpretation of certain data, to the exclusion of other possible interpretations. I usually suggest that reading several reviews of a subject is a good starting point and acts as a useful pre-selection of interesting/relevant papers. However, they also reflect the authors' prejudices, so I also very strongly recommend that the students go back to the original papers to see if they agree with the interpretation of the actual data. My two highest cited publications (about two quite different subjects within my field) came from the realisation that two opposing (and hotly defended) interpretations of similar data were both correct, depending on circumstances. Of course that made two enemies rather than just one, such is life.

Allo V Psycho
Catbabel
Posts: 734
Joined: Sat Nov 16, 2019 8:18 am

Re: There is too much science for science to work properly

Post by Allo V Psycho » Fri Nov 12, 2021 2:09 pm

Something I regret is the size of conferences. When I started out, there would about 200 attendees at a conference, and just one main session. So you sat through talks on a range of of different topics, and, in addition to getting a grasp of the whole field, unexpected cross-fertilisations seemed quite common. Now if there are 3000 attendees, they are sorted into quite narrow topics in multiple sessions. Plenaries help to some extent, but sometimes it is the less well known bit of odd information that brings about a paradigm shift.

(Anecdote irrelevant to the thread)

The history of a field also seems less accessible. I remember (as a post-doc in 1983) going to my boss and saying "I've just been reading this curious paper published in 1927, what do you think?" He said "why not ask the author, he has a room in the basement". And, true enough, down among the central heating pipes, like something from Gormenghast, was a wizened old guy.....

Only in Oxford.

Post Reply