So, trying to write the introduction to a paper, I asked it a specific question. It gave me 10 references, with a short summary for each showing how it answered my question really well. Then I checked these by reading all the full articles.Research questions are often detailed. Answering them can require looking at a topic from multiple angles. Today, we are introducing Scholar Labs, an AI powered Scholar search that is designed to help you answer detailed research questions.
It analyzes your question to identify its key topics, aspects and relationships. It then searches for all of them on Scholar, and evaluates the results to identify papers that answer the overall research question. For each paper, it provides a summary description of how the paper helps answer the question. And includes all the familiar Scholar features that you depend upon.
One was accurate, and the content matched the short summary. Seven were irrelevant: they were about different aspects of the topic, but not the one I was asking about, despite the short summary Scholar Labs had presented saying they were. Two were actively wrong - they were about the same topic, but actually said the opposite of what Scholar Labs said.
First of all, I have wasted a morning doing this. I would have found the one relevant article more quickly by hand searching, and not had to find, download and read another 9.
Second, though, if I had been tempted to take its word (which looked perfectly convincing) without checking, and had cited them using the summaries they said, I could have corrupted the literature. And I think it means when I am refereeing other peoples' papers, I have to check every single reference just in case they have been using Scholar Labs as a short cut. Yes, this would be wrong of them, but I know how much pressure researchers are under, and of course, another REF is looming for UK researchers. My employer has the stupid requirement that all research active staff produce one 4* paper every single year. Now over my long career, I have done this on average, but it varies year by year, with sometimes two or even three years going by after a successful grant application, before a bunch of papers emerged in a single year as we wrote up the project findings (where the papers interact with each other and can't really be done separately).
I use AI LLMs every day, and often find them very helpful. I love NotebookLM for turning a paper into a podcast. I'm no Luddite, but Scholar Labs looks like a real hazard to me in this context.
