Dominic Cummings.
This is the answer to all "who made the decision" questions.
The government passing the buck, say it ain’t so, Joe!Sciolus wrote: ↑Wed Aug 26, 2020 8:19 amMaybe I've missed it, but the key question I haven't seen asked or answered is: Who made the decision that the top priority should be avoiding grade inflation? That is one of the key decisions that caused the mess (the other being sticking to the original timetable). Since it's a policy decision presumably it was made by government, not Ofqual.
To be fair, I think this is where any process would be likely to start. Allowing major grade inflation means that University courses would have a much larger number of applicants who reached their unconditional offers. Some courses (such as Medicine and Dentistry) have external caps on their possible numbers, and in any case, the Government had introduced its own caps on most courses. Massive increases in students reaching their offers therefore poses serious problems, which echo down the years. And other employers than just HE take note of school level qualifications. (Of course, this bad place is exactly where we have ended up, in the most chaotic way possible: but it isn't a bad starting principle in itself). Two other options were to have a better algorithm (which I think was possible), or to run exams in a modified format.Sciolus wrote: ↑Wed Aug 26, 2020 8:19 amMaybe I've missed it, but the key question I haven't seen asked or answered is: Who made the decision that the top priority should be avoiding grade inflation? That is one of the key decisions that caused the mess (the other being sticking to the original timetable). Since it's a policy decision presumably it was made by government, not Ofqual.
You say 'that stupid', but I don't think it was stupidity; I suspect it was a political calculation. An algorithm which would have been better grounded on a population basis would have been to accept the CAG grades for state school kids and downgraded the CAGs for selective and private school kids. That might not have appealed politically. And with any algorithm you are applying a population-based calculation to individuals, as I said up-thread.Sciolus wrote: ↑Wed Aug 26, 2020 9:26 amDamn you for being fair! I actually agree with that. The thing is, though, that they came up with several algorithms and this was the one that they (who?) preferred. That choice is a policy decision and the minister should carry the can for the backlash.
We also need much more information about the alternatives that were considered. When I wrote "Go to a sh.t school? We'll make sure you get a sh.t grade", I actually thought I was misunderstanding the algorithm and they wouldn't be that stupid. But they were. Did they consider a school-specific optimism factor derived from previous years' predicted grades, for instance? Or a national-scale adjustment factor? What are the pros and cons of those approaches? Where was the discussion and how was the decision taken?
I think this whole mess highlights the problem with relying too much on algorithms (of the heuristic variety) - it's all fine and dandy until you use them for something that really matters.Allo V Psycho wrote: ↑Wed Aug 26, 2020 11:37 amAn algorithm which would have been better grounded on a population basis would have been to accept the CAG grades for state school kids and downgraded the CAGs for selective and private school kids. That might not have appealed politically. And with any algorithm you are applying a population-based calculation to individuals, as I said up-thread.
Indeed, and as far as I can tell from talking to those who mark them, I can't see they'd be that difficult to convert. It still comes back to "What the hell had they been doing all that time?"Allo V Psycho wrote: ↑Wed Aug 26, 2020 11:37 amYou say 'that stupid', but I don't think it was stupidity; I suspect it was a political calculation. An algorithm which would have been better grounded on a population basis would have been to accept the CAG grades for state school kids and downgraded the CAGs for selective and private school kids. That might not have appealed politically. And with any algorithm you are applying a population-based calculation to individuals, as I said up-thread.Sciolus wrote: ↑Wed Aug 26, 2020 9:26 amDamn you for being fair! I actually agree with that. The thing is, though, that they came up with several algorithms and this was the one that they (who?) preferred. That choice is a policy decision and the minister should carry the can for the backlash.
We also need much more information about the alternatives that were considered. When I wrote "Go to a sh.t school? We'll make sure you get a sh.t grade", I actually thought I was misunderstanding the algorithm and they wouldn't be that stupid. But they were. Did they consider a school-specific optimism factor derived from previous years' predicted grades, for instance? Or a national-scale adjustment factor? What are the pros and cons of those approaches? Where was the discussion and how was the decision taken?
A better approach from a psychometric perspective would have been to use every scrap of summative data available for each student (e.g. any course work in that year, GCSE results etc. People would still have complained that 'I was going to improve this year' but in my experience the emotional energy round this one can be overcome.
Better still (or 'less worse') would have been to run a modified summative process slightly later than the usual exam time. Perhaps Computer Adaptive Testing: certainly objectively scored (e.g. MCQs and Very Short Answers): multiple test forms, with test equating. Yes, there would have been squeals at the time, but the Germans ran exams, and seemed to manage...
(Just seen that I should have typed 'conditional offers' not unconditional ones in my previous post).
An algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
Recommended.The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
More or less had a bit on it.bmforre wrote: ↑Tue Sep 08, 2020 7:18 pmSerious opinion piece on this in NYTimes
Meredith Broussard is an artificial intelligence researcher at New York University.
She writesAn algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
High-achieving, low-income students were hit particularly hard.
Recommended.The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
I really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.bmforre wrote: ↑Tue Sep 08, 2020 7:18 pmSerious opinion piece on this in NYTimes
Meredith Broussard is an artificial intelligence researcher at New York University.
She writesAn algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
High-achieving, low-income students were hit particularly hard.
Recommended.The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Exactly - to look at the GCSE results of the class, but not the individual is perverse. Which is why I mentioned that in the link to the More or Less discussion on itAllo V Psycho wrote: ↑Tue Sep 08, 2020 7:57 pmI really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.bmforre wrote: ↑Tue Sep 08, 2020 7:18 pmSerious opinion piece on this in NYTimes
Meredith Broussard is an artificial intelligence researcher at New York University.
She writesAn algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
High-achieving, low-income students were hit particularly hard.
Recommended.The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Yes.Allo V Psycho wrote: ↑Tue Sep 08, 2020 7:57 pmI really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.bmforre wrote: ↑Tue Sep 08, 2020 7:18 pmSerious opinion piece on this in NYTimes
Meredith Broussard is an artificial intelligence researcher at New York University.
She writesAn algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
High-achieving, low-income students were hit particularly hard.
Recommended.The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
Are you sure it was as competent as that? The results don't seem to suggest that.Bird on a Fire wrote: ↑Wed Sep 09, 2020 12:35 amThe algorithms are almost certainly tuned to give a particular, desirable result.
Even if they're not, at some point a set of algorithms were selected between, almost certainly in part by "sanity-checking" against expected results.
I'm not sure how you would expect that to be taken into account. Maybe by adding a penalty to poorer students on the grounds that they would have fallen further behind due to fewer resources for studying at home?I'd be hugely surprised if nobody in an Ofqual meeting had mentioned how different socio-economic circumstances would affect different students during lockdown,
Oh yes. It's not the fork that's the problem here, it's the prick with the fork.Allo V Psycho wrote: ↑Tue Sep 08, 2020 7:57 pmI really don't understand why 'algorithms' are being blamed, and I would have expected a computer scientist to understand this. I'll say again: the problem is that population-based data was applied to individuals. If an algorithm had been derived from an individual student's previous performance, and used to predict their future performance, it would have given a better result than using population level data to predict individual performance. It wouldn't be perfect: people aren't completely predictable. But the best guide to future performance is (individual) past performance.bmforre wrote: ↑Tue Sep 08, 2020 7:18 pmSerious opinion piece on this in NYTimes
Meredith Broussard is an artificial intelligence researcher at New York University.
She writesAn algorithm used to “predict” students’ grades was based on an array of student information, including past performance by students in each school.When Algorithms Give Real Students Imaginary Grades
In-person final exams were canceled for thousands of students this spring, so computers stepped in — to disastrous effect.
High-achieving, low-income students were hit particularly hard.
Recommended.The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall ...
So why did Galois die in a duel? Surely he had the better algorithm?secret squirrel wrote: ↑Wed Sep 09, 2020 7:09 am"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."
I think Galois had the opposite of an algorithm.bmforre wrote: ↑Wed Sep 09, 2020 7:58 amSo why did Galois die in a duel? Surely he had the better algorithm?secret squirrel wrote: ↑Wed Sep 09, 2020 7:09 am"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."
Did he not get the grade he was expecting like?
I thought he did the coursework option.bmforre wrote: ↑Wed Sep 09, 2020 7:58 amSo why did Galois die in a duel? Surely he had the better algorithm?secret squirrel wrote: ↑Wed Sep 09, 2020 7:09 am"The only thing that can stop a bad guy with an algorithm is a good guy with an algorithm."