I agree with your conclusions completely, and your paper is still terrible

Get your science fix here: research, quackery, activism and all the rest
User avatar
Fishnut
Snowbonk
Posts: 546
Joined: Mon Nov 11, 2019 1:15 pm

I agree with your conclusions completely, and your paper is still terrible

Post by Fishnut » Fri Jun 19, 2020 7:29 pm

Retraction Watch have an interesting article on the logic of asking for retractions of papers whose conclusions are sound but whose data are shoddy. They give a devil's advocate position that it's best to let sleeping dogs lie and that by picking apart papers you agree with it lends credence to the idea that no-one knows what the hell is going on.

They then list four reasons for asking for these papers to be retracted, by saying that if you don't:
1) it reduces the permissible standard of evidence for future papers
2) it gives ammunition to those who don't agree with the conclusions
3) it harms public perception
4) we deserve to have the most accurate science possible

It's a short but packed piece and well worth a read.

User avatar
Matatouille
Clardic Fug
Posts: 184
Joined: Mon Nov 11, 2019 5:26 pm

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Matatouille » Sat Jun 20, 2020 6:32 am

I love Retraction Watch. They're well worth following on social media.

I'm no scientist, but I thought it would be pretty obvious that sh.t science should be retracted even if the conclusions seem "right". A better paper can then come along in a bit* and demonstrate if the thing is is true or not. The better paper's authors don't have to worry their cotton socks about publishing duplication studies, overturning the zeitgeist etc. Is it really seen as much more complicated than that?

*weeks**, months, years, decades perhaps depending upon field.
**in the context of COVID research perhaps.

secret squirrel
Fuzzable
Posts: 270
Joined: Wed Nov 13, 2019 12:42 pm

Re: I agree with your conclusions completely, and your paper is still terrible

Post by secret squirrel » Sat Jun 20, 2020 10:07 am

An interesting variation of this problem happens in maths. In theory, a flawed proof is no proof at all, but in reality there's a big grey area of technically flawed but fixable proofs. This ranges from gaps and errors that most people giving the paper a careful reading will be able to fix as they go, to completely misguided arguments that need to be replaced with new ones. Most papers probably have a few of the first kind of errors, and at the other extreme, papers whose whole approach is flawed need to be retracted, even if the main result turns out to be true. There are also awkward cases in the middle where the paper has lots of good and correct new ideas, but one part of the main argument is badly flawed and not easy to correct. Typically if this is discovered the authors will publish an erratum covering the gap, but sometimes they can't solve it and it takes another author or authors to step in and fill it. Unfortunately, there are lots of examples in the literature of papers where it's kind of folklore knowledge that some of the results are either not true, or not proved properly. There are also very rare cases where there ends up being disagreement about whether a proof makes sense at all (see for example the ongoing saga of Mochizuki's 'proof' of the ABC conjecture).

Looking back into history there are also several occasions where someone noticed and 'proved' an important result, but it took a long time for a correct argument (or sometimes even a fully correct statement) to be found. This doesn't really happen these days as mathematicians have more or less settled on a standard of rigour based, in principle, on well defined axioms, which was not the case prior to the mid 19th century or so. These days, barring extreme cases like the ABC conjecture mentioned above, mathematicians generally agree with each other about what it takes to prove a mathematical statement, though there is a small movement towards checking complicated proofs using computers (this is very much a work in progress as the technology isn't there yet for this to be feasible for complicated arguments without immense effort).

kerrya1
Bank Butt
Posts: 32
Joined: Tue Nov 12, 2019 11:13 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by kerrya1 » Thu Jun 25, 2020 12:57 pm

I have a lot of thoughts about this as someone who works at a Russell Group university trying to get researchers to properly manage and publish their data.

However, I am currently arguing with a senior PI who thinks they can meet funder requirements for data sharing by keeping all the data on a portable HD in his desk drawer. I will try to respond properly once I have beaten him into submission.

User avatar
Woodchopper
Dorkwood
Posts: 1417
Joined: Sat Oct 12, 2019 9:05 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Woodchopper » Thu Jun 25, 2020 1:58 pm

kerrya1 wrote:
Thu Jun 25, 2020 12:57 pm
I have a lot of thoughts about this as someone who works at a Russell Group university trying to get researchers to properly manage and publish their data.

However, I am currently arguing with a senior PI who thinks they can meet funder requirements for data sharing by keeping all the data on a portable HD in his desk drawer. I will try to respond properly once I have beaten him into submission.
Perhaps you could swap his hard drive with a blank one ...

User avatar
Gfamily
Dorkwood
Posts: 1352
Joined: Mon Nov 11, 2019 1:00 pm

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Gfamily » Thu Jun 25, 2020 2:28 pm

kerrya1 wrote:
Thu Jun 25, 2020 12:57 pm
I have a lot of thoughts about this as someone who works at a Russell Group university trying to get researchers to properly manage and publish their data.

However, I am currently arguing with a senior PI who thinks they can meet funder requirements for data sharing by keeping all the data on a portable HD in his desk drawer. I will try to respond properly once I have beaten him into submission.
I recall hearing about a an IT Resilience Consultant who used to keep a chainsaw in the back of his car.
If he was ever told "We have resilience because we have backup servers", he would ask to be excused, get his chainsaw fire it up and ask "Right, which one should I work on?"

It may have been apocryphal.
My avatar was a scientific result that was later found to be 'mistaken' - I rarely claim to be 100% correct

kerrya1
Bank Butt
Posts: 32
Joined: Tue Nov 12, 2019 11:13 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by kerrya1 » Thu Jun 25, 2020 3:16 pm

Woodchopper wrote:
Thu Jun 25, 2020 1:58 pm
kerrya1 wrote:
Thu Jun 25, 2020 12:57 pm
I have a lot of thoughts about this as someone who works at a Russell Group university trying to get researchers to properly manage and publish their data.

However, I am currently arguing with a senior PI who thinks they can meet funder requirements for data sharing by keeping all the data on a portable HD in his desk drawer. I will try to respond properly once I have beaten him into submission.
Perhaps you could swap his hard drive with a blank one ...
Pre-lockdown if I went to a researcher's office and they had passwords written down on show anywhere I would remove them while their backs were turned. Unfortunately the opportunities for direct intervention are now much reduced. One of my favourite pictures to show when I do training is this one.

Image

The building on fire is the old School of Informatics at Edinburgh Uni. It was completely gutted by a massive fire in 2011, a lot of researchers lost everything.

User avatar
Woodchopper
Dorkwood
Posts: 1417
Joined: Sat Oct 12, 2019 9:05 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Woodchopper » Thu Jun 25, 2020 3:27 pm

kerrya1 wrote:
Thu Jun 25, 2020 3:16 pm
Woodchopper wrote:
Thu Jun 25, 2020 1:58 pm
kerrya1 wrote:
Thu Jun 25, 2020 12:57 pm
I have a lot of thoughts about this as someone who works at a Russell Group university trying to get researchers to properly manage and publish their data.

However, I am currently arguing with a senior PI who thinks they can meet funder requirements for data sharing by keeping all the data on a portable HD in his desk drawer. I will try to respond properly once I have beaten him into submission.
Perhaps you could swap his hard drive with a blank one ...
Pre-lockdown if I went to a researcher's office and they had passwords written down on show anywhere I would remove them while their backs were turned. Unfortunately the opportunities for direct intervention are now much reduced. One of my favourite pictures to show when I do training is this one.

Image

The building on fire is the old School of Informatics at Edinburgh Uni. It was completely gutted by a massive fire in 2011, a lot of researchers lost everything.
A very long time ago I worked in the archives of an Edinburgh insurance company. The backups were located about 10 miles away.

Squeak
Fuzzable
Posts: 302
Joined: Tue Dec 03, 2019 6:27 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Squeak » Fri Jun 26, 2020 12:20 am

kerrya1 wrote:
Thu Jun 25, 2020 12:57 pm
I have a lot of thoughts about this as someone who works at a Russell Group university trying to get researchers to properly manage and publish their data.

However, I am currently arguing with a senior PI who thinks they can meet funder requirements for data sharing by keeping all the data on a portable HD in his desk drawer. I will try to respond properly once I have beaten him into submission.
Let me guess. Anyone who wants to use the data just has to email him? And they will know to email him because they've read his papers and are so desparate to reanalyse the data that they will hunt down his email address and write to him. As opposed to searching a (meta)data repository for all the well structured metadata records that describe datasets that are likely to be useful to them. Or even, heaven forbid, having the data standardised and aggregated into a system that allows multiple datasets to be merged and subsetted to answer particular questions.

My favourite (ahem) data sharing method is the one adopted by Antarctic active seismic surveyors (i.e. the ones who set off blasts from ships that may or may not upset whales). They have a network of 14 libraries around the world that you can visit in person and view data on CD-ROMS that may not leave the building. If you want to use the data, you have to email the person who collected it and humbly beg their beneficence. It sort of meets the Antarctic Treaty rule that says all observations should be freely shared...

kerrya1
Bank Butt
Posts: 32
Joined: Tue Nov 12, 2019 11:13 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by kerrya1 » Fri Jun 26, 2020 8:52 am

Squeak wrote:
Fri Jun 26, 2020 12:20 am
kerrya1 wrote:
Thu Jun 25, 2020 12:57 pm
I have a lot of thoughts about this as someone who works at a Russell Group university trying to get researchers to properly manage and publish their data.

However, I am currently arguing with a senior PI who thinks they can meet funder requirements for data sharing by keeping all the data on a portable HD in his desk drawer. I will try to respond properly once I have beaten him into submission.
Let me guess. Anyone who wants to use the data just has to email him? And they will know to email him because they've read his papers and are so desparate to reanalyse the data that they will hunt down his email address and write to him. As opposed to searching a (meta)data repository for all the well structured metadata records that describe datasets that are likely to be useful to them. Or even, heaven forbid, having the data standardised and aggregated into a system that allows multiple datasets to be merged and subsetted to answer particular questions.

My favourite (ahem) data sharing method is the one adopted by Antarctic active seismic surveyors (i.e. the ones who set off blasts from ships that may or may not upset whales). They have a network of 14 libraries around the world that you can visit in person and view data on CD-ROMS that may not leave the building. If you want to use the data, you have to email the person who collected it and humbly beg their beneficence. It sort of meets the Antarctic Treaty rule that says all observations should be freely shared...
That pretty much sums it up. He has now agreed to deposit it in the appropriate disciplinary data repository, create a metadata record for the datasets in the University CRIS, and include a properly structured citation in all published outputs. He agreed to this when I pointed out that:
1. The paper he was writing for a particular journal (he has no doubts iit will be accepted) requires underpinning data to be placed in a public repository,
2. His research funder will reject any future grant applications if he can demonstrate compliance with their outputs sharing mandate
3. I will hunt him down and beat him with a stick if he fails to comply because I'm sick and tired of his b.llsh.t.

Now I just need to follow-up every week to make sure he actually bl..dy does it.

Squeak
Fuzzable
Posts: 302
Joined: Tue Dec 03, 2019 6:27 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Squeak » Fri Jun 26, 2020 9:04 am

kerrya1 wrote:
Fri Jun 26, 2020 8:52 am

That pretty much sums it up. He has now agreed to deposit it in the appropriate disciplinary data repository, create a metadata record for the datasets in the University CRIS, and include a properly structured citation in all published outputs. He agreed to this when I pointed out that:
1. The paper he was writing for a particular journal (he has no doubts iit will be accepted) requires underpinning data to be placed in a public repository,
2. His research funder will reject any future grant applications if he can demonstrate compliance with their outputs sharing mandate
3. I will hunt him down and beat him with a stick if he fails to comply because I'm sick and tired of his b.llsh.t.

Now I just need to follow-up every week to make sure he actually bl..dy does it.
I really, really dislike a system that makes data managers play cop on this stuff. The incentives are all wrong if the people who should be helping you be an awesome scientist instead feel the need to beat you with a stick to get minimum effort out of you. He should be beating down your door with a stick to help him get his paper accepted, a successful new grant, and a whole wodge of new citations and possible collaborations. Given the cost of collecting data vs the cost of properly sharing it, it should be a no-brainer for funding agencies and institutions to properly incentivise good data sharing practices. :evil:

Adept
Navel Tan
Posts: 7
Joined: Mon Nov 11, 2019 8:11 pm

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Adept » Sun Jun 28, 2020 1:00 am

This isn't just a problem in academia. I've spent my career, such as it is,in various bits of the public sector, and getting people who work for different parts of the same organisation to share data is an uphill battle even when it's clearly to the benefit of everyone concerned and will result in us making better decisions.

Going back to the OP, I think the article is bang on, if a methodologically unsound study, or one that is based on poor data reaches the right conclusion, this is surely more by luck than judgment, and accepting (or failing to retract) such papers is only going to lead to a lowering of standards, and conclusions that ought to be challenged early on becoming accepted wisdom.

User avatar
Brightonian
Fuzzable
Posts: 247
Joined: Mon Nov 11, 2019 3:16 pm

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Brightonian » Sun Jun 28, 2020 10:15 am

Adept wrote:
Sun Jun 28, 2020 1:00 am
This isn't just a problem in academia. I've spent my career, such as it is,in various bits of the public sector, and getting people who work for different parts of the same organisation to share data is an uphill battle even when it's clearly to the benefit of everyone concerned and will result in us making better decisions.
In my last job there was a department-wide wiki. As my job required me to document, and show to others, what had been done, I moved most of my documentation onto the wiki.

And as others had documentation which would have been useful to me and others, I naively expected them to put their stuff onto the wiki. But no, they continued to keep their own documents, sometimes very nicely finished PDFs that must have taken ages to produce, in their own parts of the file system inaccessible to other sections. OK, many people don't quite get wikis or how to create/edit pages, and some document types are difficult to maintain on wiki pages, but in most situations they could have put stuff on the wiki with little trouble.

kerrya1
Bank Butt
Posts: 32
Joined: Tue Nov 12, 2019 11:13 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by kerrya1 » Tue Jun 30, 2020 11:24 am

Squeak wrote:
Fri Jun 26, 2020 9:04 am
kerrya1 wrote:
Fri Jun 26, 2020 8:52 am

That pretty much sums it up. He has now agreed to deposit it in the appropriate disciplinary data repository, create a metadata record for the datasets in the University CRIS, and include a properly structured citation in all published outputs. He agreed to this when I pointed out that:
1. The paper he was writing for a particular journal (he has no doubts iit will be accepted) requires underpinning data to be placed in a public repository,
2. His research funder will reject any future grant applications if he can demonstrate compliance with their outputs sharing mandate
3. I will hunt him down and beat him with a stick if he fails to comply because I'm sick and tired of his b.llsh.t.

Now I just need to follow-up every week to make sure he actually bl..dy does it.
I really, really dislike a system that makes data managers play cop on this stuff. The incentives are all wrong if the people who should be helping you be an awesome scientist instead feel the need to beat you with a stick to get minimum effort out of you. He should be beating down your door with a stick to help him get his paper accepted, a successful new grant, and a whole wodge of new citations and possible collaborations. Given the cost of collecting data vs the cost of properly sharing it, it should be a no-brainer for funding agencies and institutions to properly incentivise good data sharing practices. :evil:
I've been doing this for >10 years and I can count the number of PIs I've worked with who really understand the importance of Data Management & Sharing and want to do it well on my fingers and toes. I focus many of my efforts on PGRs, trying to instill them with good practice and that seems to be working, but it will be years before they are PIs and in the meantime they need to avoid picking up the bad habits of their PIs.

For things to improve we need the following:
  • 1) Research Funders to properly enforce their policies and mandates
    2) Research Funders and Institutions to properly fund all aspects of RDM, big lumps of computer storage are fine but we also need proper training and time to implement good practice to be funded explicitly
    3) Institutions and Government to move the evaluation/career advancement systems away from high impact papers or research income earned towards rewarding good practice in all areas of research, including making publicly funded data open available where possible, funding replication and extension studies, supporting the publication of negative result papers with associated data
    4) The entire culture of research to move to one where you must "show your working" - this is the entire premise of the Open Science push currently.
So, there are plenty of real researchers on here - off you go and DO BETTER!

User avatar
Woodchopper
Dorkwood
Posts: 1417
Joined: Sat Oct 12, 2019 9:05 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Woodchopper » Tue Jun 30, 2020 11:55 am

kerrya1 wrote:
Tue Jun 30, 2020 11:24 am
Squeak wrote:
Fri Jun 26, 2020 9:04 am
kerrya1 wrote:
Fri Jun 26, 2020 8:52 am

That pretty much sums it up. He has now agreed to deposit it in the appropriate disciplinary data repository, create a metadata record for the datasets in the University CRIS, and include a properly structured citation in all published outputs. He agreed to this when I pointed out that:
1. The paper he was writing for a particular journal (he has no doubts iit will be accepted) requires underpinning data to be placed in a public repository,
2. His research funder will reject any future grant applications if he can demonstrate compliance with their outputs sharing mandate
3. I will hunt him down and beat him with a stick if he fails to comply because I'm sick and tired of his b.llsh.t.

Now I just need to follow-up every week to make sure he actually bl..dy does it.
I really, really dislike a system that makes data managers play cop on this stuff. The incentives are all wrong if the people who should be helping you be an awesome scientist instead feel the need to beat you with a stick to get minimum effort out of you. He should be beating down your door with a stick to help him get his paper accepted, a successful new grant, and a whole wodge of new citations and possible collaborations. Given the cost of collecting data vs the cost of properly sharing it, it should be a no-brainer for funding agencies and institutions to properly incentivise good data sharing practices. :evil:
I've been doing this for >10 years and I can count the number of PIs I've worked with who really understand the importance of Data Management & Sharing and want to do it well on my fingers and toes. I focus many of my efforts on PGRs, trying to instill them with good practice and that seems to be working, but it will be years before they are PIs and in the meantime they need to avoid picking up the bad habits of their PIs.

For things to improve we need the following:
  • 1) Research Funders to properly enforce their policies and mandates
    2) Research Funders and Institutions to properly fund all aspects of RDM, big lumps of computer storage are fine but we also need proper training and time to implement good practice to be funded explicitly
    3) Institutions and Government to move the evaluation/career advancement systems away from high impact papers or research income earned towards rewarding good practice in all areas of research, including making publicly funded data open available where possible, funding replication and extension studies, supporting the publication of negative result papers with associated data
    4) The entire culture of research to move to one where you must "show your working" - this is the entire premise of the Open Science push currently.
So, there are plenty of real researchers on here - off you go and DO BETTER!
The basic problem is that the way that funding and journal publication is organized PIs are in intense competition with each other. I agree that shifting career progression away from bringing in big grants and high impact journals would help. But they are still there.

User avatar
Bird on a Fire
Stummy Beige
Posts: 2597
Joined: Fri Oct 11, 2019 5:05 pm
Location: with the birds

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Bird on a Fire » Tue Jun 30, 2020 4:02 pm

Yeah, the incentives for researchers to share data aren't always that clear. It's a lot of very tedious work to help other people you don't know, who are often in direct competition with you for grants and jobs etc., use your data to further their careers. At least in my field, each data point can reflect hours and hours of work, so having to 'give it away' after a single publication would be pretty frustrating.

There are basically zero carrots in the current system, and the sticks aren't generally very effective either, so I'm not surprised uptake is poor tbh.

I suspect there will be a generational difference - ECRs coming up in the last 10 or so years are a lot more familiar with and accepting of the arguments around open data, at least in my (limited) experience.
now I'm falling asleep and she's calling acab

Squeak
Fuzzable
Posts: 302
Joined: Tue Dec 03, 2019 6:27 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Squeak » Tue Jun 30, 2020 11:28 pm

The scientists I know who are really on board with data sharing and do it well tend to be absolute evangelists for data sharing because they can see that they collect citations and collaboration opportunities. And that includes biologists. Check out the MEOP seal data, which was put together by and for scientists and only got professional data managers involved later, for a nice example. Everyone involved in setting up that collaboration has seen huge reputational boosts. (Admittedly, the photos of seals with trackers glued to their foreheads are pretty popular with senior bosses too.)

The ones who get dragged into writing a metadata record and dumping a non-standardized file into an institutional data centre seem to be much more cynical about the benefits of data sharing, which is unsurprising because that data will be hard to find and hard to reuse so most potential citers and collaborators won't bother.

Somewhat like Kerry, I'd like to see data publication and reuse metrics genuinely count for grants and job and promotion opportunities, in much the same way that papers work. Someone who collected a data point that has helped answer five, ten, or 1000 research questions is someone whose contribution to science should be rewarded more than someone who's collected a data point to answer one or two questions. At least on that metric. The data hoarder might be brilliant in other ways, of course.

But the culture change is slow. Even in the Antarctic, where the Treaty has legally required data sharing since the 1950s, I've had conversations with youngish scientists who are absolutely horrified to learn that they have these legal obligations.

Allo V Psycho
Clardic Fug
Posts: 184
Joined: Sat Nov 16, 2019 8:18 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Allo V Psycho » Thu Jul 02, 2020 8:14 am

Bird on a Fire wrote:
Tue Jun 30, 2020 4:02 pm
Yeah, the incentives for researchers to share data aren't always that clear. It's a lot of very tedious work to help other people you don't know, who are often in direct competition with you for grants and jobs etc., use your data to further their careers. At least in my field, each data point can reflect hours and hours of work, so having to 'give it away' after a single publication would be pretty frustrating.

There are basically zero carrots in the current system, and the sticks aren't generally very effective either, so I'm not surprised uptake is poor tbh.

I suspect there will be a generational difference - ECRs coming up in the last 10 or so years are a lot more familiar with and accepting of the arguments around open data, at least in my (limited) experience.
What BoaF said.

The institutional pressures on researchers (in HE) are currently about meeting REF targets for publications, impact, and environment. This is what they are getting really (i.e. 'we're considering sacking you') beaten by a stick for. Kerry is describing a kind of personal social pressure exerted on researchers, and I'd be concerned that it could lead to confrontation, which is likely to make everyone even more stressed and unhappy than they already are.

I think a solution might be to organise the process so that it all becomes integrated from the beginning: your ethics application has to contain the data archiving structure, so the grant application automatically does so, and the data is collected and stored during the research in a compatible way so that 'a button just has to be pressed' at the end. We often have to get NHS ethics (which the University then accepts automatically) so if it was part of the NHS REC requirements, it would have to be met in advance.

Even aside from really genuine pressures, there is a hierarchy of fun in research. Devising experiments - great fun. Carrying out experiments - sometimes fun, sometimes really challenging. Analysing data (that shows something cool) - best fun in the entire Universe. Writing data up for publication - tedious non-fun. Formatting the draft for submission to a particular journal- extremely tedious non-fun. Archiving data - worse than that*.

*I'm going to confess that if my most recently submitted paper is published, it will say "Data available from the corresponding author on reasonable request", in line with the journal style.

Going back to the OP - I'm sure I commented on this on some other thread. It IS the data, not the 'conclusion' that matters. If your data is unreliable, it is irrelevant that the conclusion 'corresponds to the currently accepted paradigm', because that paradigm may well shift at some point, and then bad data delays and muddies the paradigm shift.

kerrya1
Bank Butt
Posts: 32
Joined: Tue Nov 12, 2019 11:13 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by kerrya1 » Thu Jul 02, 2020 9:40 am

Allo V Psycho wrote:
Thu Jul 02, 2020 8:14 am
I think a solution might be to organise the process so that it all becomes integrated from the beginning: your ethics application has to contain the data archiving structure, so the grant application automatically does so, and the data is collected and stored during the research in a compatible way so that 'a button just has to be pressed' at the end. We often have to get NHS ethics (which the University then accepts automatically) so if it was part of the NHS REC requirements, it would have to be met in advance.

The process ought already to be integrated with the grant application process, the UKRI funders (EPSRC excepted), Wellcome Trust, Cancer Research UK, Horizon 2020, + others all require a Data Management Plan / Outputs Management Plan which should clearly state how the data will be managed during the grant period, where and when the data will be archived, and on what terms it will be accessible to others. This has been the Case for well over 5, in some case 10 years now.
Allo V Psycho wrote:
Thu Jul 02, 2020 8:14 am
Even aside from really genuine pressures, there is a hierarchy of fun in research. Devising experiments - great fun. Carrying out experiments - sometimes fun, sometimes really challenging. Analysing data (that shows something cool) - best fun in the entire Universe. Writing data up for publication - tedious non-fun. Formatting the draft for submission to a particular journal- extremely tedious non-fun. Archiving data - worse than that*.
I am quite aware that most researchers don't find the process of research data management and sharing fun, but if they would just follow the guidance provided by funders and institutions, use the tools already available to them, and recognise that doing this will have benefits to them as well as others then it would take up much less of their time and energy.
Allo V Psycho wrote:
Thu Jul 02, 2020 8:14 am
*I'm going to confess that if my most recently submitted paper is published, it will say "Data available from the corresponding author on reasonable request", in line with the journal style.
I hate you and hope that your lab burns down taking your all your data with it! #justkidding?

Allo V Psycho
Clardic Fug
Posts: 184
Joined: Sat Nov 16, 2019 8:18 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Allo V Psycho » Thu Jul 02, 2020 2:26 pm

kerrya1 wrote:
Thu Jul 02, 2020 9:40 am
Allo V Psycho wrote:
Thu Jul 02, 2020 8:14 am
I think a solution might be to organise the process so that it all becomes integrated from the beginning: your ethics application has to contain the data archiving structure, so the grant application automatically does so, and the data is collected and stored during the research in a compatible way so that 'a button just has to be pressed' at the end. We often have to get NHS ethics (which the University then accepts automatically) so if it was part of the NHS REC requirements, it would have to be met in advance.

The process ought already to be integrated with the grant application process, the UKRI funders (EPSRC excepted), Wellcome Trust, Cancer Research UK, Horizon 2020, + others all require a Data Management Plan / Outputs Management Plan which should clearly state how the data will be managed during the grant period, where and when the data will be archived, and on what terms it will be accessible to others. This has been the Case for well over 5, in some case 10 years now.
Allo V Psycho wrote:
Thu Jul 02, 2020 8:14 am
Even aside from really genuine pressures, there is a hierarchy of fun in research. Devising experiments - great fun. Carrying out experiments - sometimes fun, sometimes really challenging. Analysing data (that shows something cool) - best fun in the entire Universe. Writing data up for publication - tedious non-fun. Formatting the draft for submission to a particular journal- extremely tedious non-fun. Archiving data - worse than that*.
I am quite aware that most researchers don't find the process of research data management and sharing fun, but if they would just follow the guidance provided by funders and institutions, use the tools already available to them, and recognise that doing this will have benefits to them as well as others then it would take up much less of their time and energy.
Allo V Psycho wrote:
Thu Jul 02, 2020 8:14 am
*I'm going to confess that if my most recently submitted paper is published, it will say "Data available from the corresponding author on reasonable request", in line with the journal style.
I hate you and hope that your lab burns down taking your all your data with it! #justkidding?
:lol: :lol: :lol:
Thank you for your restraint in not adding "and you in it"...

I haven't applied for a UKRI Grant in well over 10 years due to a change in my research direction, so I didn't know that: thanks for the info. I've had a number of non-UKRI grants, though, none of which had this kind of requirement for data handling. So, genuine question: what about qualitative research, which is about half of my publications now? How would that be stored?

kerrya1
Bank Butt
Posts: 32
Joined: Tue Nov 12, 2019 11:13 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by kerrya1 » Thu Jul 02, 2020 6:36 pm

Allo V Psycho wrote:
Thu Jul 02, 2020 2:26 pm
:lol: :lol: :lol:
Thank you for your restraint in not adding "and you in it"...

I haven't applied for a UKRI Grant in well over 10 years due to a change in my research direction, so I didn't know that: thanks for the info. I've had a number of non-UKRI grants, though, none of which had this kind of requirement for data handling. So, genuine question: what about qualitative research, which is about half of my publications now? How would that be stored?
Absolutely qualitative data can and should be archived, whether it can be shared is slightly more complicated. If you were a PI that I was advising then I'd be asking these questions to be able to advise you correctly:

*How is your research funded? Does your funder (or institution) have any relevant policies?

*Is the data sensitive (personal data, commercially sensitive, etc.)?
If it is personal data, can it be adequately anonymised to permit open sharing or will access control be required?
If it is commercially sensitive what is the position of the commercial or industry partners/data providers/research funder regarding data sharing?

*Format and scale of data - text, audio, video? MB, GB, TB?

*Policy on under-pining data of journal you wish to publish in?

*Is good quality metadata and documentation to support the data and make it Findable, Accessible, Interoperable, & Reuseable (FAIR) available

Depending on the answers to these questions I would recommend a suitable Data Repository and deposit strategy.

Of course, had you written a Data Management Plan at the outset with support from my team then you'd already know all the answers, your metadata and documentation would be excellent and you would know which data repository you were going to use.

I recommend this for writing your DMP https://dmponline.dcc.ac.uk/

Squeak
Fuzzable
Posts: 302
Joined: Tue Dec 03, 2019 6:27 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Squeak » Fri Jul 03, 2020 7:46 am

Allo, I work with data managers who specialise in social data from indigenous communities. They build community-owned systems to store all sorts of qualitative and quantitative data from place name pronunciations to old stories about land use, to observations of wildlife and landscape change. They care for data with lots of varying controls and cultural sensitivities and, because the data are properly curated and documented, it's much easier to ensure it's used respectfully into the future. It's not a simple process, particularly because of the history of exploitation and knowledge extraction from these communities, but it can be done and done extremely well.

User avatar
Woodchopper
Dorkwood
Posts: 1417
Joined: Sat Oct 12, 2019 9:05 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Woodchopper » Fri Jul 03, 2020 7:59 am

I’m currently the PI of a project which it partly based upon semi-structured interviews. The project doesn’t deal with sensitive data (eg medical, crime etc) and we aren’t asking for personal data. We are asking for their experiences of working in a specific role.

The regulator’s policy is that all the source material has to be deleted. We’ll delete the recordings of interviews after 30 days and transcripts and other notes after the end of the project. The rationale is that interviews can’t be anonymized and we aren’t supposed to keep non-anonymized data.

It’s possible to get an exemption so I’ve asked to be able to keep the email addresses of interviewees so that I can send them a copy of the publications. Hopefully they’ll agree.

User avatar
Woodchopper
Dorkwood
Posts: 1417
Joined: Sat Oct 12, 2019 9:05 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by Woodchopper » Fri Jul 03, 2020 8:01 am

Squeak wrote:
Fri Jul 03, 2020 7:46 am
Allo, I work with data managers who specialise in social data from indigenous communities. They build community-owned systems to store all sorts of qualitative and quantitative data from place name pronunciations to old stories about land use, to observations of wildlife and landscape change. They care for data with lots of varying controls and cultural sensitivities and, because the data are properly curated and documented, it's much easier to ensure it's used respectfully into the future. It's not a simple process, particularly because of the history of exploitation and knowledge extraction from these communities, but it can be done and done extremely well.
That looks very cool.

kerrya1
Bank Butt
Posts: 32
Joined: Tue Nov 12, 2019 11:13 am

Re: I agree with your conclusions completely, and your paper is still terrible

Post by kerrya1 » Fri Jul 03, 2020 9:40 am

Woodchopper wrote:
Fri Jul 03, 2020 8:01 am
Squeak wrote:
Fri Jul 03, 2020 7:46 am
Allo, I work with data managers who specialise in social data from indigenous communities. They build community-owned systems to store all sorts of qualitative and quantitative data from place name pronunciations to old stories about land use, to observations of wildlife and landscape change. They care for data with lots of varying controls and cultural sensitivities and, because the data are properly curated and documented, it's much easier to ensure it's used respectfully into the future. It's not a simple process, particularly because of the history of exploitation and knowledge extraction from these communities, but it can be done and done extremely well.
That looks very cool.
Excellent, sounds similar to this work at Edinburgh https://media.ed.ac.uk/playlist/dedicat ... 0_bpqm3pzm

Do they use Omeka or another platform?

Post Reply