Scrutable

I use Google Scholar a lot, and now it features on the landing page "Scholar Labs".

Research questions are often detailed. Answering them can require looking at a topic from multiple angles. Today, we are introducing Scholar Labs, an AI powered Scholar search that is designed to help you answer detailed research questions.

It analyzes your question to identify its key topics, aspects and relationships. It then searches for all of them on Scholar, and evaluates the results to identify papers that answer the overall research question. For each paper, it provides a summary description of how the paper helps answer the question. And includes all the familiar Scholar features that you depend upon.

So, trying to write the introduction to a paper, I asked it a specific question. It gave me 10 references, with a short summary for each showing how it answered my question really well. Then I checked these by reading all the full articles.

One was accurate, and the content matched the short summary. Seven were irrelevant: they were about different aspects of the topic, but not the one I was asking about, despite the short summary Scholar Labs had presented saying they were. Two were actively wrong - they were about the same topic, but actually said the opposite of what Scholar Labs said.

First of all, I have wasted a morning doing this. I would have found the one relevant article more quickly by hand searching, and not had to find, download and read another 9.

Second, though, if I had been tempted to take its word (which looked perfectly convincing) without checking, and had cited them using the summaries they said, I could have corrupted the literature. And I think it means when I am refereeing other peoples' papers, I have to check every single reference just in case they have been using Scholar Labs as a short cut. Yes, this would be wrong of them, but I know how much pressure researchers are under, and of course, another REF is looming for UK researchers. My employer has the stupid requirement that all research active staff produce one 4* paper every single year. Now over my long career, I have done this on average, but it varies year by year, with sometimes two or even three years going by after a successful grant application, before a bunch of papers emerged in a single year as we wrote up the project findings (where the papers interact with each other and can't really be done separately).
I use AI LLMs every day, and often find them very helpful. I love NotebookLM for turning a paper into a podcast. I'm no Luddite, but Scholar Labs looks like a real hazard to me in this context.

You have to check every reference when reviewing papers anyway, because there's a fair chance they were generated by ChatGPT, and thus entirely hallucinated.

Yes, that's what I'm saying - I had hoped that something called Scholar Lab from Google Scholar would be better. And it isn't quite hallucinating references in the style I'm used to from ChatGPT, since all the references are to real papers. They just don't say what the summary says. In my experience the ChatGPT hallucinated papers tend to have invented titles or journals and are therefore fairly easy to identify. The Scholar Lab papers are all genuine papers, but I have to read them carefully before I can be sure that they do not in fact say what the Scholar Lab summary says they say.
Typically I get a request to referee a paper about every week, I decline about half, so undertake about two a month, typically it takes me about half a day to referee a paper properly, mostly checking the data says what they say it says (influenced by our Steamy member of the forum). There is no way on earth I can read and check 50 references as well, so now I fear I cannot referee any, which is potentially not great for the field.

With respect to references, I check a few random ones when I am reviewing a paper, unless a statement strikes me as being worth checking into. A paper I reviewed recently, contained a statement I considered untrue, so I thought I would check the references associated with that statement, only to find the references did not support it. That led me to checking all the references in the Introduction and only a single reference supported the statement made. This was the author's previous paper. Needless to say, I recommended rejection.