AI and references to published papers

[Gfamily]

An interesting podcast episode* from How Many Geese in which they asked an AI text generator to write a section for a podcast.

Came up with an interesting story (bio sciences and ecology related) and was able to provide references.
But - there's a catch.
Spoiler:

The story, scenario and references were completely invented, but used real researcher names and institutions

* Episode 'Now That's What I Call Nature' from 16 May 23.

[bob sterman]

Not sure we need a spoiler warning? Thought this was pretty widely known??

But I'll go along with it...

Spoiler:

We've found this in student work at our university. Many AI engines fabricate lists of references with plausible combinations of author names, article and journal titles, even plausible pagination. Part of the "hallucination" problem they have.

[Imrael]

Similarly for legal stuff apparently.

BBC acticle

[FlammableFlower]

There are other AIs that are better at this apparently and are much better at supplying real references - these are subscription rather than free.

We've seen it now in a few places in submissions by students. Now it could just be the case of us seeing the obvious stuff and missing the really good ones... but in the main you can spot it in the language - it's generally too polished and "highfalutin" even by undergrad standards where they are often trying to sound "scientific" by using big words. Mainly it's the references, which are bl..dy awful: fake, or if real they are poor. One example - a student cited a reference in a blatantly ChatGPT-d paragraph, apparently a global study about the pandemic and indoor air quality, but it actually was a study from 2012 in some specific US counties.

We had a play around and asked it about different academic staff - it frequently got their area of research wrong (!) and just invented papers by them.

Another thing - they don't seem to be very good at a sustaining things over length. Could possibly pass muster in something that is 500 words long and someone wasn't paying attention, but not so good for longer and more complex pieces.

ETA - some of my colleagues have got quite spooked and panicked by this... I'm currently more sanguine.

[dyqik]

There's also the problem that it invents software libraries when asked to provide code for common programming tasks.

People have noticed this, and started generating malware packages with those names, that get automatically downloaded, installed and run when the code is implemented by a programmer.

[Chris Preston]

Yes we have seen much of the same for student reports. I am somewhat worried when students twig that if they do some additional work they can hide the AI pretty well. We are now getting training on how to set assignments that are hard to do with AI.

I received one assignment where the references were all real, if rather broad reviews, but what struck me about the reference list was the Accessed on dates went all the way back to 2018, with none from 2023, for an assignment set in 2023.

The lazy students who typically plagiarise are going to leave the AI tells in their work.

[dyqik]

Yes we have seen much of the same for student reports. I am somewhat worried when students twig that if they do some additional work they can hide the AI pretty well. We are now getting training on how to set assignments that are hard to do with AI.

I received one assignment where the references were all real, if rather broad reviews, but what struck me about the reference list was the Accessed on dates went all the way back to 2018, with none from 2023, for an assignment set in 2023.

The lazy students who typically plagiarise are going to leave the AI tells in their work.

The best version of an AI-proof assignment I've heard of asked students to generate an AI essay in response to a prompt of their choice, and then find and explain the errors in it.

[bjn]

Current LLM don't "hallucinate" the wrong answers and get correct answers "right", they hallucinate everything, it's just that sometimes they hallucinate the correct answer. The best way to think of any request to an LLM is not "please tell me the answer to this question", but rather "please tell me what an answer to this question might look like". When it gets it right, it just stumbled upon it. Stochastic parrots are one of the better descriptions I've found. Some things that are very well known (eg: height of Mount Everest) it is very likely to get right, the more obscure it gets, the less likely. All depending on the input data that was used to generate the LLM.

Apparently there's an LLM that used 4chan as input. Unsurprisingly, its responses are truly horrendous.

[Chris Preston]

The Junior Ps were here the other evening and were using Chat GTP to get recipes using strange collections of ingredients. At one point they decided that MSG had to be an ingredient, so asked for a recipe using a cup of MSG. Chat GTP wrote it couldn't do that, because it would be too unhealthy.

[Woodchopper]

Yes we have seen much of the same for student reports. I am somewhat worried when students twig that if they do some additional work they can hide the AI pretty well. We are now getting training on how to set assignments that are hard to do with AI.

Yes, I've played around with GPT a bit. It would be foolish to copy and paste. But as far as I can see someone who is intelligent but lazy could spend a couple of hours editing a CHAT GPT output rather than a couple of days reading material and writing an essay from scratch. Problem is that the lazy one won't understand the subject nearly as well, if at all.

[Woodchopper]

Yes we have seen much of the same for student reports. I am somewhat worried when students twig that if they do some additional work they can hide the AI pretty well. We are now getting training on how to set assignments that are hard to do with AI.

Yes, I've played around with GPT a bit. It would be foolish to copy and paste. But as far as I can see someone who is intelligent but lazy could spend a couple of hours editing a CHAT GPT output rather than a couple of days reading material and writing an essay from scratch. Problem is that the lazy one won't understand the subject nearly as well, if at all.

For example, what Chat GPT seems to be good at is structuring an argument and listing the most important points. If someone is writing from scratch then getting that structure right can take some time and the author needs to first understand the subject.

[Imrael]

There's also the problem that it invents software libraries when asked to provide code for common programming tasks.

People have noticed this, and started generating malware packages with those names, that get automatically downloaded, installed and run when the code is implemented by a programmer.

A colleague I was talking to about the issue found this. ALthough its aimed at use in software dev I cant help thinking it would apply more broadly.
The poisoning of ChatGPT