Page 7 of 7

Re: A-levels mess

Posted: Wed Aug 26, 2020 8:26 am
by lpm
Sciolus wrote:
Wed Aug 26, 2020 8:19 am
Who made the decision
Dominic Cummings.

This is the answer to all "who made the decision" questions.

Re: A-levels mess

Posted: Wed Aug 26, 2020 8:28 am
by Little waster
Sciolus wrote:
Wed Aug 26, 2020 8:19 am
Maybe I've missed it, but the key question I haven't seen asked or answered is: Who made the decision that the top priority should be avoiding grade inflation? That is one of the key decisions that caused the mess (the other being sticking to the original timetable). Since it's a policy decision presumably it was made by government, not Ofqual.
The government passing the buck, say it ain’t so, Joe!

https://www.theguardian.com/society/202 ... e-rejected

Re: A-levels mess

Posted: Wed Aug 26, 2020 8:45 am
by Allo V Psycho
Sciolus wrote:
Wed Aug 26, 2020 8:19 am
Maybe I've missed it, but the key question I haven't seen asked or answered is: Who made the decision that the top priority should be avoiding grade inflation? That is one of the key decisions that caused the mess (the other being sticking to the original timetable). Since it's a policy decision presumably it was made by government, not Ofqual.
To be fair, I think this is where any process would be likely to start. Allowing major grade inflation means that University courses would have a much larger number of applicants who reached their unconditional offers. Some courses (such as Medicine and Dentistry) have external caps on their possible numbers, and in any case, the Government had introduced its own caps on most courses. Massive increases in students reaching their offers therefore poses serious problems, which echo down the years. And other employers than just HE take note of school level qualifications. (Of course, this bad place is exactly where we have ended up, in the most chaotic way possible: but it isn't a bad starting principle in itself). Two other options were to have a better algorithm (which I think was possible), or to run exams in a modified format.

Re: A-levels mess

Posted: Wed Aug 26, 2020 9:26 am
by Sciolus
Damn you for being fair! I actually agree with that. The thing is, though, that they came up with several algorithms and this was the one that they (who?) preferred. That choice is a policy decision and the minister should carry the can for the backlash.

We also need much more information about the alternatives that were considered. When I wrote "Go to a sh.t school? We'll make sure you get a sh.t grade", I actually thought I was misunderstanding the algorithm and they wouldn't be that stupid. But they were. Did they consider a school-specific optimism factor derived from previous years' predicted grades, for instance? Or a national-scale adjustment factor? What are the pros and cons of those approaches? Where was the discussion and how was the decision taken?

Re: A-levels mess

Posted: Wed Aug 26, 2020 11:37 am
by Allo V Psycho
Sciolus wrote:
Wed Aug 26, 2020 9:26 am
Damn you for being fair! I actually agree with that. The thing is, though, that they came up with several algorithms and this was the one that they (who?) preferred. That choice is a policy decision and the minister should carry the can for the backlash.

We also need much more information about the alternatives that were considered. When I wrote "Go to a sh.t school? We'll make sure you get a sh.t grade", I actually thought I was misunderstanding the algorithm and they wouldn't be that stupid. But they were. Did they consider a school-specific optimism factor derived from previous years' predicted grades, for instance? Or a national-scale adjustment factor? What are the pros and cons of those approaches? Where was the discussion and how was the decision taken?
You say 'that stupid', but I don't think it was stupidity; I suspect it was a political calculation. An algorithm which would have been better grounded on a population basis would have been to accept the CAG grades for state school kids and downgraded the CAGs for selective and private school kids. That might not have appealed politically. And with any algorithm you are applying a population-based calculation to individuals, as I said up-thread.

A better approach from a psychometric perspective would have been to use every scrap of summative data available for each student (e.g. any course work in that year, GCSE results etc. People would still have complained that 'I was going to improve this year' but in my experience the emotional energy round this one can be overcome.

Better still (or 'less worse') would have been to run a modified summative process slightly later than the usual exam time. Perhaps Computer Adaptive Testing: certainly objectively scored (e.g. MCQs and Very Short Answers): multiple test forms, with test equating. Yes, there would have been squeals at the time, but the Germans ran exams, and seemed to manage...

(Just seen that I should have typed 'conditional offers' not unconditional ones in my previous post).

Re: A-levels mess

Posted: Wed Aug 26, 2020 1:42 pm
by bob sterman
Allo V Psycho wrote:
Wed Aug 26, 2020 11:37 am
An algorithm which would have been better grounded on a population basis would have been to accept the CAG grades for state school kids and downgraded the CAGs for selective and private school kids. That might not have appealed politically. And with any algorithm you are applying a population-based calculation to individuals, as I said up-thread.
I think this whole mess highlights the problem with relying too much on algorithms (of the heuristic variety) - it's all fine and dandy until you use them for something that really matters.

E.g. if Facebook or Google use a heuristic algorithm that decides to show me adverts for power tools when I'm actually in the market for some eyelash extensions - no real harm done.

However, when the Metropolitan Police use an algorithm to decide whether to subject me to a stop-and-search, or DUFFQUAL use one to decide whether I can go to university - things are rather different.

Re: A-levels mess

Posted: Wed Aug 26, 2020 4:50 pm
by bagpuss
What I can't understand about the process (one of many things that make no sense, admittedly) is that, if the algorithm was supposed to produce grades that were in line with previous years' results at any given school/exam centre, why were so many schools reporting that their results were significantly lower than previous years?

I haven't seen overall stats, I'm sure there are some around, but I heard a number of schools reporting that their grades were lower than any recent prior years. For example, someone from Marlow Grammar (so a school that you might actually expect to have benefitted) reported that their awarded grades were more than 10% down on the worst year from the last 5 years.

So, regardless of whether this is a desirable thing to achieve, why did the algorithm not even manage to achieve its stated aim of ensuring schools got grades consistent with previous years? Have I missed something that explained this?

Re: A-levels mess

Posted: Wed Aug 26, 2020 6:08 pm
by FlammableFlower
Allo V Psycho wrote:
Wed Aug 26, 2020 11:37 am
Sciolus wrote:
Wed Aug 26, 2020 9:26 am
Damn you for being fair! I actually agree with that. The thing is, though, that they came up with several algorithms and this was the one that they (who?) preferred. That choice is a policy decision and the minister should carry the can for the backlash.

We also need much more information about the alternatives that were considered. When I wrote "Go to a sh.t school? We'll make sure you get a sh.t grade", I actually thought I was misunderstanding the algorithm and they wouldn't be that stupid. But they were. Did they consider a school-specific optimism factor derived from previous years' predicted grades, for instance? Or a national-scale adjustment factor? What are the pros and cons of those approaches? Where was the discussion and how was the decision taken?
You say 'that stupid', but I don't think it was stupidity; I suspect it was a political calculation. An algorithm which would have been better grounded on a population basis would have been to accept the CAG grades for state school kids and downgraded the CAGs for selective and private school kids. That might not have appealed politically. And with any algorithm you are applying a population-based calculation to individuals, as I said up-thread.

A better approach from a psychometric perspective would have been to use every scrap of summative data available for each student (e.g. any course work in that year, GCSE results etc. People would still have complained that 'I was going to improve this year' but in my experience the emotional energy round this one can be overcome.

Better still (or 'less worse') would have been to run a modified summative process slightly later than the usual exam time. Perhaps Computer Adaptive Testing: certainly objectively scored (e.g. MCQs and Very Short Answers): multiple test forms, with test equating. Yes, there would have been squeals at the time, but the Germans ran exams, and seemed to manage...

(Just seen that I should have typed 'conditional offers' not unconditional ones in my previous post).
Indeed, and as far as I can tell from talking to those who mark them, I can't see they'd be that difficult to convert. It still comes back to "What the hell had they been doing all that time?"

Re: A-levels mess

Posted: Tue Sep 08, 2020 7:18 pm
by bmforre
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.

Re: A-levels mess

Posted: Tue Sep 08, 2020 7:32 pm
by jimbob
bmforre wrote:
Tue Sep 08, 2020 7:18 pm
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.
More or less had a bit on it.

The school's past A-level results were included. And if the cohort had significantly different GCSE results from normal, they were taken account of (not sure how that was supposed to work in 6th form colleges) but the statement was made that the student's *individual* GCSE results were not taken into account.

6:33 into this https://www.bbc.co.uk/sounds/play/p08nznp9

Just realised my signature is possibly apt.

Re: A-levels mess

Posted: Tue Sep 08, 2020 7:57 pm
by Allo V Psycho
bmforre wrote:
Tue Sep 08, 2020 7:18 pm
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.
I really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.

Re: A-levels mess

Posted: Tue Sep 08, 2020 8:38 pm
by jimbob
Allo V Psycho wrote:
Tue Sep 08, 2020 7:57 pm
bmforre wrote:
Tue Sep 08, 2020 7:18 pm
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.
I really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.
Exactly - to look at the GCSE results of the class, but not the individual is perverse. Which is why I mentioned that in the link to the More or Less discussion on it

Re: A-levels mess

Posted: Tue Sep 08, 2020 10:14 pm
by FlammableFlower
Allo V Psycho wrote:
Tue Sep 08, 2020 7:57 pm
bmforre wrote:
Tue Sep 08, 2020 7:18 pm
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.
I really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.
Yes.

An algorithm in and of itself is not necessarily bad... that particular algorithm used by the government was bl..dy awful.

Re: A-levels mess

Posted: Wed Sep 09, 2020 12:35 am
by Bird on a Fire
The algorithms are almost certainly tuned to give a particular, desirable result.

Even if they're not, at some point a set of algorithms were selected between, almost certainly in part by "sanity-checking" against expected results.

I'd be hugely surprised if nobody in an Ofqual meeting had mentioned how different socio-economic circumstances would affect different students during lockdown, and I'm sure many journalists' and legal departments are already working on FOIing those minutes. When they're leaked they won't tell us anything we didn't already know.

Re: A-levels mess

Posted: Wed Sep 09, 2020 1:39 am
by Millennie Al
Bird on a Fire wrote:
Wed Sep 09, 2020 12:35 am
The algorithms are almost certainly tuned to give a particular, desirable result.

Even if they're not, at some point a set of algorithms were selected between, almost certainly in part by "sanity-checking" against expected results.
Are you sure it was as competent as that? The results don't seem to suggest that.
I'd be hugely surprised if nobody in an Ofqual meeting had mentioned how different socio-economic circumstances would affect different students during lockdown,
I'm not sure how you would expect that to be taken into account. Maybe by adding a penalty to poorer students on the grounds that they would have fallen further behind due to fewer resources for studying at home?

Re: A-levels mess

Posted: Wed Sep 09, 2020 1:50 am
by bmforre
I do agree with the criticisms of the AI researcher's criticism.

I find that those with power and influence over grade-setting accept very feeble justifications for choices and decisions making things easy for themselves. Never mind candidates being graded this year.

Re: A-levels mess

Posted: Wed Sep 09, 2020 6:17 am
by basementer
Allo V Psycho wrote:
Tue Sep 08, 2020 7:57 pm
bmforre wrote:
Tue Sep 08, 2020 7:18 pm
Serious opinion piece on this in NYTimes

Meredith Broussard is an artificial intelligence researcher at New York University.
She writes
When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.
High-achieving, low-income students were hit particularly hard.
The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Recommended.
I really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.
Oh yes. It's not the fork that's the problem here, it's the prick with the fork.

Re: A-levels mess

Posted: Wed Sep 09, 2020 7:09 am
by secret squirrel
"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."

Re: A-levels mess

Posted: Wed Sep 09, 2020 7:58 am
by bmforre
secret squirrel wrote:
Wed Sep 09, 2020 7:09 am
"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."
So why did Galois die in a duel? Surely he had the better algorithm?

Re: A-levels mess

Posted: Wed Sep 09, 2020 9:33 am
by Sciolus
It's poor bl..dy old Muḥammad ibn Mūsā al-Khwārizmī I feel sorry for.

Re: A-levels mess

Posted: Wed Sep 09, 2020 12:53 pm
by secret squirrel
bmforre wrote:
Wed Sep 09, 2020 7:58 am
secret squirrel wrote:
Wed Sep 09, 2020 7:09 am
"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."
So why did Galois die in a duel? Surely he had the better algorithm?
I think Galois had the opposite of an algorithm.

Re: A-levels mess

Posted: Wed Sep 09, 2020 2:15 pm
by Little waster
Sciolus wrote:
Wed Sep 09, 2020 9:33 am
It's poor bl..dy old Muḥammad ibn Mūsā al-Khwārizmī I feel sorry for.
Did he not get the grade he was expecting like? :(








;)

Re: A-levels mess

Posted: Wed Sep 09, 2020 5:40 pm
by Gfamily
bmforre wrote:
Wed Sep 09, 2020 7:58 am
secret squirrel wrote:
Wed Sep 09, 2020 7:09 am
"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."
So why did Galois die in a duel? Surely he had the better algorithm?
I thought he did the coursework option.