A-levels mess

Discussions about serious topics, for serious people
Post Reply
User avatar
lpm
After Pie
Posts: 1828
Joined: Mon Nov 11, 2019 1:05 pm
Location: INJECT HIM WITH BLEACH

Re: A-levels mess

Post by lpm » Wed Aug 26, 2020 8:26 am

Sciolus wrote:
Wed Aug 26, 2020 8:19 am
Who made the decision
Dominic Cummings.

This is the answer to all "who made the decision" questions.
I'll miss him after he's died in the pandemic

User avatar
Little waster
Catbabel
Posts: 882
Joined: Tue Nov 12, 2019 12:35 am
Location: About 1 inch behind my eyes

Re: A-levels mess

Post by Little waster » Wed Aug 26, 2020 8:28 am

Sciolus wrote:
Wed Aug 26, 2020 8:19 am
Maybe I've missed it, but the key question I haven't seen asked or answered is: Who made the decision that the top priority should be avoiding grade inflation? That is one of the key decisions that caused the mess (the other being sticking to the original timetable). Since it's a policy decision presumably it was made by government, not Ofqual.
The government passing the buck, say it ain’t so, Joe!

https://www.theguardian.com/society/202 ... e-rejected
People who use Godwin's Law are worse than Hitler.

Allo V Psycho
Clardic Fug
Posts: 210
Joined: Sat Nov 16, 2019 8:18 am

Re: A-levels mess

Post by Allo V Psycho » Wed Aug 26, 2020 8:45 am

Sciolus wrote:
Wed Aug 26, 2020 8:19 am
Maybe I've missed it, but the key question I haven't seen asked or answered is: Who made the decision that the top priority should be avoiding grade inflation? That is one of the key decisions that caused the mess (the other being sticking to the original timetable). Since it's a policy decision presumably it was made by government, not Ofqual.
To be fair, I think this is where any process would be likely to start. Allowing major grade inflation means that University courses would have a much larger number of applicants who reached their unconditional offers. Some courses (such as Medicine and Dentistry) have external caps on their possible numbers, and in any case, the Government had introduced its own caps on most courses. Massive increases in students reaching their offers therefore poses serious problems, which echo down the years. And other employers than just HE take note of school level qualifications. (Of course, this bad place is exactly where we have ended up, in the most chaotic way possible: but it isn't a bad starting principle in itself). Two other options were to have a better algorithm (which I think was possible), or to run exams in a modified format.

User avatar
Sciolus
Fuzzable
Posts: 338
Joined: Mon Nov 11, 2019 6:42 pm

Re: A-levels mess

Post by Sciolus » Wed Aug 26, 2020 9:26 am

Damn you for being fair! I actually agree with that. The thing is, though, that they came up with several algorithms and this was the one that they (who?) preferred. That choice is a policy decision and the minister should carry the can for the backlash.

We also need much more information about the alternatives that were considered. When I wrote "Go to a sh.t school? We'll make sure you get a sh.t grade", I actually thought I was misunderstanding the algorithm and they wouldn't be that stupid. But they were. Did they consider a school-specific optimism factor derived from previous years' predicted grades, for instance? Or a national-scale adjustment factor? What are the pros and cons of those approaches? Where was the discussion and how was the decision taken?

Allo V Psycho
Clardic Fug
Posts: 210
Joined: Sat Nov 16, 2019 8:18 am

Re: A-levels mess

Post by Allo V Psycho » Wed Aug 26, 2020 11:37 am

Sciolus wrote:
Wed Aug 26, 2020 9:26 am
Damn you for being fair! I actually agree with that. The thing is, though, that they came up with several algorithms and this was the one that they (who?) preferred. That choice is a policy decision and the minister should carry the can for the backlash.

We also need much more information about the alternatives that were considered. When I wrote "Go to a sh.t school? We'll make sure you get a sh.t grade", I actually thought I was misunderstanding the algorithm and they wouldn't be that stupid. But they were. Did they consider a school-specific optimism factor derived from previous years' predicted grades, for instance? Or a national-scale adjustment factor? What are the pros and cons of those approaches? Where was the discussion and how was the decision taken?
You say 'that stupid', but I don't think it was stupidity; I suspect it was a political calculation. An algorithm which would have been better grounded on a population basis would have been to accept the CAG grades for state school kids and downgraded the CAGs for selective and private school kids. That might not have appealed politically. And with any algorithm you are applying a population-based calculation to individuals, as I said up-thread.

A better approach from a psychometric perspective would have been to use every scrap of summative data available for each student (e.g. any course work in that year, GCSE results etc. People would still have complained that 'I was going to improve this year' but in my experience the emotional energy round this one can be overcome.

Better still (or 'less worse') would have been to run a modified summative process slightly later than the usual exam time. Perhaps Computer Adaptive Testing: certainly objectively scored (e.g. MCQs and Very Short Answers): multiple test forms, with test equating. Yes, there would have been squeals at the time, but the Germans ran exams, and seemed to manage...

(Just seen that I should have typed 'conditional offers' not unconditional ones in my previous post).

User avatar
bob sterman
Fuzzable
Posts: 272
Joined: Mon Nov 11, 2019 10:25 pm

Re: A-levels mess

Post by bob sterman » Wed Aug 26, 2020 1:42 pm

Allo V Psycho wrote:
Wed Aug 26, 2020 11:37 am
An algorithm which would have been better grounded on a population basis would have been to accept the CAG grades for state school kids and downgraded the CAGs for selective and private school kids. That might not have appealed politically. And with any algorithm you are applying a population-based calculation to individuals, as I said up-thread.
I think this whole mess highlights the problem with relying too much on algorithms (of the heuristic variety) - it's all fine and dandy until you use them for something that really matters.

E.g. if Facebook or Google use a heuristic algorithm that decides to show me adverts for power tools when I'm actually in the market for some eyelash extensions - no real harm done.

However, when the Metropolitan Police use an algorithm to decide whether to subject me to a stop-and-search, or DUFFQUAL use one to decide whether I can go to university - things are rather different.

bagpuss
Snowbonk
Posts: 450
Joined: Tue Nov 12, 2019 12:10 pm

Re: A-levels mess

Post by bagpuss » Wed Aug 26, 2020 4:50 pm

What I can't understand about the process (one of many things that make no sense, admittedly) is that, if the algorithm was supposed to produce grades that were in line with previous years' results at any given school/exam centre, why were so many schools reporting that their results were significantly lower than previous years?

I haven't seen overall stats, I'm sure there are some around, but I heard a number of schools reporting that their grades were lower than any recent prior years. For example, someone from Marlow Grammar (so a school that you might actually expect to have benefitted) reported that their awarded grades were more than 10% down on the worst year from the last 5 years.

So, regardless of whether this is a desirable thing to achieve, why did the algorithm not even manage to achieve its stated aim of ensuring schools got grades consistent with previous years? Have I missed something that explained this?

FlammableFlower
Catbabel
Posts: 711
Joined: Mon Nov 11, 2019 1:22 pm

Re: A-levels mess

Post by FlammableFlower » Wed Aug 26, 2020 6:08 pm

Allo V Psycho wrote:
Wed Aug 26, 2020 11:37 am
Sciolus wrote:
Wed Aug 26, 2020 9:26 am
Damn you for being fair! I actually agree with that. The thing is, though, that they came up with several algorithms and this was the one that they (who?) preferred. That choice is a policy decision and the minister should carry the can for the backlash.

We also need much more information about the alternatives that were considered. When I wrote "Go to a sh.t school? We'll make sure you get a sh.t grade", I actually thought I was misunderstanding the algorithm and they wouldn't be that stupid. But they were. Did they consider a school-specific optimism factor derived from previous years' predicted grades, for instance? Or a national-scale adjustment factor? What are the pros and cons of those approaches? Where was the discussion and how was the decision taken?
You say 'that stupid', but I don't think it was stupidity; I suspect it was a political calculation. An algorithm which would have been better grounded on a population basis would have been to accept the CAG grades for state school kids and downgraded the CAGs for selective and private school kids. That might not have appealed politically. And with any algorithm you are applying a population-based calculation to individuals, as I said up-thread.

A better approach from a psychometric perspective would have been to use every scrap of summative data available for each student (e.g. any course work in that year, GCSE results etc. People would still have complained that 'I was going to improve this year' but in my experience the emotional energy round this one can be overcome.

Better still (or 'less worse') would have been to run a modified summative process slightly later than the usual exam time. Perhaps Computer Adaptive Testing: certainly objectively scored (e.g. MCQs and Very Short Answers): multiple test forms, with test equating. Yes, there would have been squeals at the time, but the Germans ran exams, and seemed to manage...

(Just seen that I should have typed 'conditional offers' not unconditional ones in my previous post).
Indeed, and as far as I can tell from talking to those who mark them, I can't see they'd be that difficult to convert. It still comes back to "What the hell had they been doing all that time?"

bmforre
Fuzzable
Posts: 321
Joined: Mon Nov 11, 2019 1:15 pm
Location: Trondheim

Re: A-levels mess

Post by bmforre » Tue Sep 08, 2020 7:18 pm

Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.

User avatar
jimbob
After Pie
Posts: 1601
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: A-levels mess

Post by jimbob » Tue Sep 08, 2020 7:32 pm

bmforre wrote:
Tue Sep 08, 2020 7:18 pm
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.
More or less had a bit on it.

The school's past A-level results were included. And if the cohort had significantly different GCSE results from normal, they were taken account of (not sure how that was supposed to work in 6th form colleges) but the statement was made that the student's *individual* GCSE results were not taken into account.

6:33 into this https://www.bbc.co.uk/sounds/play/p08nznp9

Just realised my signature is possibly apt.
Have you considered stupidity as an explanation

Allo V Psycho
Clardic Fug
Posts: 210
Joined: Sat Nov 16, 2019 8:18 am

Re: A-levels mess

Post by Allo V Psycho » Tue Sep 08, 2020 7:57 pm

bmforre wrote:
Tue Sep 08, 2020 7:18 pm
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.
I really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.

User avatar
jimbob
After Pie
Posts: 1601
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: A-levels mess

Post by jimbob » Tue Sep 08, 2020 8:38 pm

Allo V Psycho wrote:
Tue Sep 08, 2020 7:57 pm
bmforre wrote:
Tue Sep 08, 2020 7:18 pm
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.
I really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.
Exactly - to look at the GCSE results of the class, but not the individual is perverse. Which is why I mentioned that in the link to the More or Less discussion on it
Have you considered stupidity as an explanation

FlammableFlower
Catbabel
Posts: 711
Joined: Mon Nov 11, 2019 1:22 pm

Re: A-levels mess

Post by FlammableFlower » Tue Sep 08, 2020 10:14 pm

Allo V Psycho wrote:
Tue Sep 08, 2020 7:57 pm
bmforre wrote:
Tue Sep 08, 2020 7:18 pm
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.
I really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.
Yes.

An algorithm in and of itself is not necessarily bad... that particular algorithm used by the government was bl..dy awful.

User avatar
Bird on a Fire
Stummy Beige
Posts: 3772
Joined: Fri Oct 11, 2019 5:05 pm
Location: nadir of brie

Re: A-levels mess

Post by Bird on a Fire » Wed Sep 09, 2020 12:35 am

The algorithms are almost certainly tuned to give a particular, desirable result.

Even if they're not, at some point a set of algorithms were selected between, almost certainly in part by "sanity-checking" against expected results.

I'd be hugely surprised if nobody in an Ofqual meeting had mentioned how different socio-economic circumstances would affect different students during lockdown, and I'm sure many journalists' and legal departments are already working on FOIing those minutes. When they're leaked they won't tell us anything we didn't already know.
“One of the penalties of an ecological education is that one lives alone in a world of wounds.”
― Aldo Leopold, A Sand County Almanac (1949)

#ShowYourStripes

Millennie Al
Fuzzable
Posts: 262
Joined: Mon Mar 16, 2020 4:02 am

Re: A-levels mess

Post by Millennie Al » Wed Sep 09, 2020 1:39 am

Bird on a Fire wrote:
Wed Sep 09, 2020 12:35 am
The algorithms are almost certainly tuned to give a particular, desirable result.

Even if they're not, at some point a set of algorithms were selected between, almost certainly in part by "sanity-checking" against expected results.
Are you sure it was as competent as that? The results don't seem to suggest that.
I'd be hugely surprised if nobody in an Ofqual meeting had mentioned how different socio-economic circumstances would affect different students during lockdown,
I'm not sure how you would expect that to be taken into account. Maybe by adding a penalty to poorer students on the grounds that they would have fallen further behind due to fewer resources for studying at home?
Covid-19 - Don't catch it: don't spread it.

bmforre
Fuzzable
Posts: 321
Joined: Mon Nov 11, 2019 1:15 pm
Location: Trondheim

Re: A-levels mess

Post by bmforre » Wed Sep 09, 2020 1:50 am

I do agree with the criticisms of the AI researcher's criticism.

I find that those with power and influence over grade-setting accept very feeble justifications for choices and decisions making things easy for themselves. Never mind candidates being graded this year.

User avatar
basementer
Snowbonk
Posts: 541
Joined: Mon Nov 11, 2019 1:03 pm
Location: Aotearoa

Re: A-levels mess

Post by basementer » Wed Sep 09, 2020 6:17 am

Allo V Psycho wrote:
Tue Sep 08, 2020 7:57 pm
bmforre wrote:
Tue Sep 08, 2020 7:18 pm
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.
I really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.
Oh yes. It's not the fork that's the problem here, it's the prick with the fork.
Money is just a substitute for luck anyway. - Tom Siddell

secret squirrel
Fuzzable
Posts: 331
Joined: Wed Nov 13, 2019 12:42 pm

Re: A-levels mess

Post by secret squirrel » Wed Sep 09, 2020 7:09 am

"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."

bmforre
Fuzzable
Posts: 321
Joined: Mon Nov 11, 2019 1:15 pm
Location: Trondheim

Re: A-levels mess

Post by bmforre » Wed Sep 09, 2020 7:58 am

secret squirrel wrote:
Wed Sep 09, 2020 7:09 am
"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."
So why did Galois die in a duel? Surely he had the better algorithm?

User avatar
Sciolus
Fuzzable
Posts: 338
Joined: Mon Nov 11, 2019 6:42 pm

Re: A-levels mess

Post by Sciolus » Wed Sep 09, 2020 9:33 am

It's poor bl..dy old Muḥammad ibn Mūsā al-Khwārizmī I feel sorry for.

secret squirrel
Fuzzable
Posts: 331
Joined: Wed Nov 13, 2019 12:42 pm

Re: A-levels mess

Post by secret squirrel » Wed Sep 09, 2020 12:53 pm

bmforre wrote:
Wed Sep 09, 2020 7:58 am
secret squirrel wrote:
Wed Sep 09, 2020 7:09 am
"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."
So why did Galois die in a duel? Surely he had the better algorithm?
I think Galois had the opposite of an algorithm.

User avatar
Little waster
Catbabel
Posts: 882
Joined: Tue Nov 12, 2019 12:35 am
Location: About 1 inch behind my eyes

Re: A-levels mess

Post by Little waster » Wed Sep 09, 2020 2:15 pm

Sciolus wrote:
Wed Sep 09, 2020 9:33 am
It's poor bl..dy old Muḥammad ibn Mūsā al-Khwārizmī I feel sorry for.
Did he not get the grade he was expecting like? :(








;)
People who use Godwin's Law are worse than Hitler.

User avatar
Gfamily
After Pie
Posts: 1743
Joined: Mon Nov 11, 2019 1:00 pm

Re: A-levels mess

Post by Gfamily » Wed Sep 09, 2020 5:40 pm

bmforre wrote:
Wed Sep 09, 2020 7:58 am
secret squirrel wrote:
Wed Sep 09, 2020 7:09 am
"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."
So why did Galois die in a duel? Surely he had the better algorithm?
I thought he did the coursework option.
My avatar was a scientific result that was later found to be 'mistaken' - I rarely claim to be 100% correct
ETA 5/8/20: I've been advised that the result was correct, it was the initial interpretation that needed to be withdrawn
Meta? I'd say so!

Post Reply