Forum analysis

[Fishnut]

Introduction
The forum has been going for a couple of months so I thought it would be interesting to have a look at how things are going. I was inspired by the discussion in the Pit which seems to have turned into a bit of a meta-discussion on how to make the public sub-forums more appealing to visitors and what constitutes an "interesting" thread. I don't want to get too in depth as to the latter but I can speak to the former from my own experiences. What attracted me to the old place and that was the well-referenced, robust, and evidence-based discussions on a wide range of subjects.

I wanted to see how we're all doing on that front and so I've had a look through the first page of threads in Nerd Lab (NL) and Weighty Matters (WM) to see how many start with links, how many replies involve links and what sort of links are being provided. I've not included Relaxation Station in this analysis because it's not public and it's not got the same intentions as NL or WM.

A reminder of the forum rules regarding these sub-forums:
NL: "Discussions about (mainly) science issues. Expect requests for evidence, challenges to assertions, and a focus on facts."

WM: "Discussions about serious topics, for serious people."

(Both quotes from here).

Based on this I would expect to see NL have lots of links and a focus on scientific literature, and WM have a decent number of links to a range of sources.

Method
I went through the first page of threads in NL and WM and looked at the first page of posts. I recorded whether the opening/original post (OP) had any links and if so of what sort, and then recorded how many links were offered in the replies and of what sorts. I also recorded how many of the posts had no links.

Results
87% of posts in NL began with a link. 78% of posts in WM began with a link. The majority of links in the OP were news articles. There were five journal articles in OPs in NL but none in WM.

The replies show a much lower frequency of links. 19% of replies in NL contain at least one link while only 10% of replies in WM contain at least one link. The majority of links are news articles (44% in both NL and WM) while journal articles accounted for 17% of links in NL and only 2% in WM.

WM is more active than NL. 63% of posts examined were in WM due to the increased number of threads that reached or exceed one page.

While NL had a greater number of links, WM had a greater number of posts with links. NL had 7 posts without any links in their replies at all, compared to only 2 in WM.

Discussion
NL is being held to a higher standard of evidence than WM, as the rules require, in as much as it has a higher proportion of threads being started with reference to external sources of information and a higher proportion of links are being offered in subsequent discussions. However, these threads often die quickly. A lot of the links are to news articles rather than journal articles. While news articles are useful at presenting information in an accessible way, providing links to any relevant original research can be useful and should be encouraged.

A lot of discussions took place that could have provided references to back up statements but did not. It's unclear if this is laziness or because it's perceived as unnecessary. If we want the forum to be seen as a place for rigorous discussion I think it is important to ask people to back up claims with evidence.

The core members of this forum have known each other for a long time and it appears that this familiarity has led to people being taken at their word. While this is an admirable trait, any visitors would have no reason to do so and may see members merely providing opinion as fact. It is therefore recommended that we as members become more demanding of posters to support their claims and statements with externally sourced material. While it is appreciated that this can take time to provide, it will provide the benefits of increased rigour and ability to fact-check.

Conclusion
EOSTFU needs to become our mantra once again.

[Stephanie]

oh my god, fishnut I love you.

this is not an evidenced post, but it's true.

[dyqik]

oh my god, fishnut I love you.

this is not an evidenced post, but it's true.

First hand experience is evidence. It says so in Naomi Oreskes new book.

[murmur]

I've certainly linked to news items or something being discussed on another website.

A snag for some of us is we don't have any sort of academic account and so cannot get access to some of the papers or evidential sources, especially when news outlets don't include links to sources (that's you I'm looking at BBC, Independent and Guardian!), so we are a bit stuck.

Also, in terms of evidence we can have difficulties: there has been a news piece which I've seen reported on the Guardian and Independent about how many referrals to CAMHS are turned down, both containing an implication that this is wrong; this is an area I know a lot about, as I've read more referrals to CAMHS than most folk and been involved in CAMHS service design; however I have little actual evidence to point out bits of these reports which are bollocks, aside from my own first hand knowledge, because so little is actually researched, recorded in any sort of sensible way without any sort of managerial fiddling (my nursey blogging alter ego wrote a very snarky piece about NHS "counting"), and even if it is, it is hard to get hold of, involving FoI requests to many organisations, etc. So, I've not bothered starting a thread about it (anyway, long-standing readers from the other place must be sick of me carping about media coverage of MH issues and services, sticking the boot into NHS management and that class of thing).

[mikeh]

Nice analysis fishnut!

(anyway, long-standing readers from the other place must be sick of me carping about media coverage of MH issues and services, sticking the boot into NHS management and that class of thing).

I'm not.

More carping the better*, I say.

* Subject to terms and conditions

[Fishnut]

I've certainly linked to news items or something being discussed on another website.

A snag for some of us is we don't have any sort of academic account and so cannot get access to some of the papers or evidential sources, especially when news outlets don't include links to sources (that's you I'm looking at BBC, Independent and Guardian!), so we are a bit stuck.

Also, in terms of evidence we can have difficulties: there has been a news piece which I've seen reported on the Guardian and Independent about how many referrals to CAMHS are turned down, both containing an implication that this is wrong; this is an area I know a lot about, as I've read more referrals to CAMHS than most folk and been involved in CAMHS service design; however I have little actual evidence to point out bits of these reports which are bollocks, aside from my own first hand knowledge, because so little is actually researched, recorded in any sort of sensible way without any sort of managerial fiddling (my nursey blogging alter ego wrote a very snarky piece about NHS "counting"), and even if it is, it is hard to get hold of, involving FoI requests to many organisations, etc. So, I've not bothered starting a thread about it (anyway, long-standing readers from the other place must be sick of me carping about media coverage of MH issues and services, sticking the boot into NHS management and that class of thing).

The 'snag' is a very good point and one I have some potential solutions to. But I just want to briefly address the area of expertise. We want expertise! Of course I can't speak for everyone but one of the things that brings me back to this place is that it's got people with a wide range of expertise who are willing to share their knowledge, for free. How amazing is that! Having you talk about mental health, or Ken McKenzie on education, or EPD on railways has taught me so much over the years and in no way am I suggesting that people stop bringing their knowledge simply because they can't find a peer-reviewed journal article to back up their claims. What I am, however, worried about is variations on the theme of "my mate down the pub says...." which seems to be cropping up a fair amount. Anecdotes from personal experience can be valuable, but they are just that, a personal experience. Argument from authority is fine if you're actually an authority on the subject, though providing references for others is also very helpful as it will enable them to learn more should they chose to. What I'm wary of is of us falling into the trap of thinking "murmur knows his sh.t about mental health so I'll accept anything he says on quantum physics without asking for evidence too".

Ok. Onto the snag.

There's two parts to this - one is finding the original research when journalists don't bother to link to it, and the other is accessing it when so much is behind paywalls. I've got some solutions for both those problems that have worked for me and which might be useful for people.

Finding the Original Research
It's highly unlikely that a story about new research is only published in a single newspaper. Often universities will write press releases that get picked up by news organisations to be published in either modified form or verbatim. This means that if the place you saw the story doesn't link to the research you can google the key words and see where else it's been reported. Sometimes those other publications will be better at linking to the research. If you don't want things to be so hit-and-miss the other options are to see if the article mentions any of the following:
- institution involved in the research
- lead author of the research
- journal the research was published in
If you have any of these then finding the paper is often quite straightforward. If the institution is mentioned you can google the name and look through their recent press releases. These generally involve a link or at least a citation to the journal article. If the author is mentioned then google them and find out their institution to get the press release. If the journal is mentioned you can go to their website and look at their latest issue. It's worth noting though that a lot of journals these days have an "online first" section, or similar, which is where they put papers that have been accepted but not given a slot yet so it won't have a volume, issue or page number. If the news article says something like "researched published today in the journal Science says...." then you'll probably find the article in the latest issue but if it's vaguer and the journal has an "online first" section then it's worth looking there if you can't see anything in the latest issue. All journals should have their contents online, so you should be able to find the title of the paper at the very least.

Finding the Full Text
Paywalls suck. Fortunately a lot of papers are being published as open access now, and some journals will make archives older than x number of years freely available too. Even if a paper is behind a paywall the abstract should still be freely available which will give a more scientific, and possibly more accurate, description than what's being published in the press. If you want to see a full paper that's behind a paywall there are several options. I use an add-on called Unpaywall which provides access to any public version of a paper. You can also look and see if the authors have published the paper to Researchgate. ArXiv.org often has pre-publications of papers in the more mathematical fields. Then there's sci-hub of course. The other options are asking people here or emailing the authors and asking for a copy. I've not had anyone say no yet. The email address of the corresponding author will be given with the abstract so just ping them a quick message asking for a copy.

If you're looking for older research Google Scholar is my go-to option. It will let you know if a PDF of the paper is freely available which is very helpful.

NB
It should be said that academic papers are not the be-all-and-end-all of evidence. We all know that crap can get published, that there are predatory journals more interested in making a buck than academic rigour, and that even when it is good work, it can go out of date very quickly. Other sources are also totally valid - the only reason I picked news and journals to single out in my analysis was that they were the most common sources being used with nothing else really coming close. I'm not saying that everyone needs to produce fully referenced posts with 10 references a piece from the last 10 years from journals with an impact factor over 5. I'm just saying that if (generic) you think something is true maybe try and find some evidence to back up your claims rather than making simple assertions.

I'm sorry if all this is teaching grandmother to suck eggs but I just wanted to make sure that people weren't avoiding linking to stuff because they didn't think they'd be able to find the right sources.

[Stephanie]

The part about finding papers might very well be useful in the future, depending on the sort of people we can get to turn up here. It would be good to have some activism going on.

[Grumble]

The part about finding papers might very well be useful in the future, depending on the sort of people we can get to turn up here. It would be good to have some activism going on.

If there’s anything that can bring more members in, it’s probably the members already here being more activist and public about belonging to here.

[murmur]

Thanks Fishnut! Oh, definitely don't ask me anything to do with any branch of physics...

Mike, be careful what you ask for: you may just have triggered a looooooong post on that topic I mentioned.

[Sciolus]

On paywalled papers, I've had a fair degree of success just by looking on the author's website.

[Allo V Psycho]

Thanks, Fishnut: an interesting and valuable OP.

There's a 'but' from my point of view. Well, maybe not as strong as 'but'.

When I post on topics in my particular areas of expertise, I can generally provide a reference for all the major statements in it. But that would take a lot of time, which is precious to me. Instead, I'd rather respond to a challenge or enquiry, and then provide the evidence. This is more economical from my point of view, but I'd be curious to hear what folks think the optimum strategy is in these kinds of circumstances.

[El Pollo Diablo]

Great analysis fishnut!

[science_fox]

The part about finding papers might very well be useful in the future, depending on the sort of people we can get to turn up here. It would be good to have some activism going on.

In addition to Fishnut's resources there's Scihub - https://sci-hub.se/ which acknowledges that it's pirating paywalled papers.

Another option is simply to ask. There's a few of us who do work at universities and you never know what might make it's way to an inbox.

(mods - can send attachements as PMs? I couldn't see a link option last time I tried).

[shpalman]

The part about finding papers might very well be useful in the future, depending on the sort of people we can get to turn up here. It would be good to have some activism going on.

In addition to Fishnut's resources there's Scihub - https://sci-hub.se/ which acknowledges that it's pirating paywalled papers.

There's a Telegram bot - @scihubot - which you should not use either because it would be illegal.

Bear in mind when you use these services, the authors of the articles receive nothing for all their hard work, and neither do the other scientists who have given up their time to perform peer review.

Of course, the authors and referees don't receive anything when you download a paper legally either.

[Ken McKenzie]

Thanks, Fishnut: an interesting and valuable OP.

There's a 'but' from my point of view. Well, maybe not as strong as 'but'.

When I post on topics in my particular areas of expertise, I can generally provide a reference for all the major statements in it. But that would take a lot of time, which is precious to me. Instead, I'd rather respond to a challenge or enquiry, and then provide the evidence. This is more economical from my point of view, but I'd be curious to hear what folks think the optimum strategy is in these kinds of circumstances.

I'm going to go further.

First, the number of links is irrelevant. I can post all the links you like but that doesn't necessarily make a post actually either good or rigorous. The quality of the links are what matters. Making an argument by - just to take an example entirely at random - posting a handful of newspaper articles and then by a notorious politically-connected of policy-based evidence is really of very little value.

Second, like other people here I'm an actual expert. Uncomfortably for me, I'm an expert in something that is politically live and on which a lot of people have Very Strong Opinions that are frequently very driven by political tribalism (and so not always amenable to evidenced argument). Like Allo, I can reference all my major statements, but it's very time-consuming. I have the additional issue that a lot of the references are likely to be to my own published research. As someone who would like to engage with the forum but would like to retain some veneer of anonymity in this adversarial age, then, candidly, there is decreasingly little incentive to take part in discussion. I can waste a lot of my time providing densely-argued expertise (for which I could, instead, be getting paid) only to be informed that because I haven't compromised my anonymity then I'm not Doing It Properly or I can just do one and let you squabble. Many people will be exposed to arguments that are just plain wrong without any real countervailing input but that will no longer be my problem. As someone who tries not to engage with arguments I don't properly understand, it is immensely frustrating that forum culture seems positively to encourage people to pontificate outside their expertise by lowering the social barriers to that.

Taking some of the rest of this to The Pit.

[Bird on a Fire]

The part about finding papers might very well be useful in the future, depending on the sort of people we can get to turn up here. It would be good to have some activism going on.

In addition to Fishnut's resources there's Scihub - https://sci-hub.se/ which acknowledges that it's pirating paywalled papers.

There's a Telegram bot - @scihubot - which you should not use either because it would be illegal.

Bear in mind when you use these services, the authors of the articles receive nothing for all their hard work, and neither do the other scientists who have given up their time to perform peer review.

Of course, the authors and referees don't receive anything when you download a paper legally either.

Well exactly - the only people making money out of scientific publishing are billionaire publishing corporations, and in some cases societies that run the journals. Some editing is remunerated, but the most time-consuming stuff is all voluntary anyway.

Authors get rewarded by engagement with their work, both by citations and increasingly by social media engagement (as quantified by e.g. Altmetrics). So if somebody has written an interesting work you're doing them more of a favour by pirating it then tweeting a link than you would be by either forking out €35 for legal access or adding to their 10,000-per-diem email deluge.

Also IME practically everyone in institutions in poorer countries already scihubs everything (myself included) as we don't have access to anything otherwise. And nor can we afford to publish open-access.

Publishing is a racket.

[plodder]

Thanks, Fishnut: an interesting and valuable OP.

There's a 'but' from my point of view. Well, maybe not as strong as 'but'.

When I post on topics in my particular areas of expertise, I can generally provide a reference for all the major statements in it. But that would take a lot of time, which is precious to me. Instead, I'd rather respond to a challenge or enquiry, and then provide the evidence. This is more economical from my point of view, but I'd be curious to hear what folks think the optimum strategy is in these kinds of circumstances.

I'm going to go further.

First, the number of links is irrelevant. I can post all the links you like but that doesn't necessarily make a post actually either good or rigorous. The quality of the links are what matters. Making an argument by - just to take an example entirely at random - posting a handful of newspaper articles and then by a notorious politically-connected of policy-based evidence is really of very little value.

Second, like other people here I'm an actual expert. Uncomfortably for me, I'm an expert in something that is politically live and on which a lot of people have Very Strong Opinions that are frequently very driven by political tribalism (and so not always amenable to evidenced argument). Like Allo, I can reference all my major statements, but it's very time-consuming. I have the additional issue that a lot of the references are likely to be to my own published research. As someone who would like to engage with the forum but would like to retain some veneer of anonymity in this adversarial age, then, candidly, there is decreasingly little incentive to take part in discussion. I can waste a lot of my time providing densely-argued expertise (for which I could, instead, be getting paid) only to be informed that because I haven't compromised my anonymity then I'm not Doing It Properly or I can just do one and let you squabble. Many people will be exposed to arguments that are just plain wrong without any real countervailing input but that will no longer be my problem. As someone who tries not to engage with arguments I don't properly understand, it is immensely frustrating that forum culture seems positively to encourage people to pontificate outside their expertise by lowering the social barriers to that.

Taking some of the rest of this to The Pit.

Not sure I follow re the "forum culture". The way to stop spouting and pointificating is to post something better, that stops the pontification in its tracks. If you're linking to your own research you don't have to say "this is me". Unless the author is McKensie, K., in which case you've outed yourself already.

[Ken McKenzie]

Not sure I follow re the "forum culture". The way to stop spouting and pointificating is to post something better, that stops the pontification in its tracks. If you're linking to your own research you don't have to say "this is me". Unless the author is McKensie, K., in which case you've outed yourself already.

A very good example of why this is a problem comes from a long time ago. I was arguing with a former poster who, in order to retain his anonymity, we will call Mr J, who also had Strong Opinions both on issues in general and on his own ability to reason them out, allied with a reluctance to acknowledge another's viewpoint. Mr J (unknowingly) cited my own work against an argument I was making. He had, of course, misinterpreted the data.

This was initially very amusing indeed but made it hard to argue against because it was difficult to explain that the author of the report meant something different to Mr J's interpretation without, at best, revealing that I knew the author well. J was, characteristically, extremely reluctant to accept he'd misinterpreted the data and that reduced my options considerably.

[Stephanie]

Tbh, I think the point is more... it would be nice to have a bit more evidence. Or a better argued position. Just some more effort, so that things are a bit more interesting and less, here's a bunch of anecdotes and some Strong Opinion that it turns out I haven't thought through at all.

[Fishnut]

Tbh, I think the point is more... it would be nice to have a bit more evidence. Or a better argued position. Just some more effort, so that things are a bit more interesting and less, here's a bunch of anecdotes and some Strong Opinion that it turns out I haven't thought through at all.

This is exactly the point. I get that it's time consuming to write properly researched and referenced posts - I'd hate to think how much time I've spent researching some of my posts, but definitely been several hours. But I don't feel that time is wasted as it allows me to get a better understanding of the subject and hopefully provide something that adds value.

No-one is being forced to contribute here. If people don't think it's worth their time then that's fine. To some extent I agree - if people are happy with anecdotes and opinions presented as facts with no-one willing to consider different points of view or evidence to the contrary then why waste time trying to write reasoned and evidenced posts? But if that's the case then why not just stick to Twitter or Reddit? Why come to a tiny little forum in the back waters of the internet? All I'm suggesting is that if we want to make this place valuable for ourselves and others maybe we should be focusing a bit more on quality over quantity. Surely it's worth a try?

[Stephanie]

I'll be honest, I don't think everyone needs to spend hours making posts. I would settle for a bit more evidence and effort.

[plodder]

Not sure I follow re the "forum culture". The way to stop spouting and pointificating is to post something better, that stops the pontification in its tracks. If you're linking to your own research you don't have to say "this is me". Unless the author is McKensie, K., in which case you've outed yourself already.

A very good example of why this is a problem comes from a long time ago. I was arguing with a former poster who, in order to retain his anonymity, we will call Mr J, who also had Strong Opinions both on issues in general and on his own ability to reason them out, allied with a reluctance to acknowledge another's viewpoint. Mr J (unknowingly) cited my own work against an argument I was making. He had, of course, misinterpreted the data.

This was initially very amusing indeed but made it hard to argue against because it was difficult to explain that the author of the report meant something different to Mr J's interpretation without, at best, revealing that I knew the author well. J was, characteristically, extremely reluctant to accept he'd misinterpreted the data and that reduced my options considerably.

“I don’t want to have to link to my own research and then find that the papers can mean something other than I wanted them to” isn’t that convincing an argument tbh. I think you’re great Ken, but ivory towers and all that.

[Ken McKenzie]

“I don’t want to have to link to my own research and then find that the papers can mean something other than I wanted them to” isn’t that convincing an argument tbh. I think you’re great Ken, but ivory towers and all that.

Seriously plodder, that wasn't the issue at all. The report he'd linked to was really very hard to interpret in the way Mr J had interpreted it because I'd explicitly written something different in it and he'd taken a pair of figures out of context, very obviously in the assumption I wouldn't be familiar with the report.


The issue was that it was hard for me to argue my own work in any way other than my own words, which I'd already done in the original report. Even someone like Mr J was likely to spot that.

[plodder]

ken just seen your detailed and obviously sincere post in the pit. pls feel free to reply there - and pls also don’t take offence, you just came across a little lofty is all

[Ken McKenzie]

ken just seen your detailed and obviously sincere post in the pit. pls feel free to reply there - and pls also don’t take offence, you just came across a little lofty is all

I am a massive elitist dicklord tbf plodder.

I find this sort of thing difficult. Obviously I am hardly unhappy to be towards the top of my field but it makes it a lot harder to piss about online with my virtual mates like I used to.