A-levels mess

Discussions about serious topics, for serious people
Post Reply
User avatar
shpalman
After Pie
Posts: 2195
Joined: Mon Nov 11, 2019 12:53 pm
Location: One step beyond

Re: mess

Post by shpalman » Fri Aug 14, 2020 10:01 am

https://www.thepoke.co.uk/2020/08/13/ga ... same-joke/

Gavin Williamson is quoted as saying that boosting exam grades 'would harm Generation Covid for life' while the words being attributed to him are a paraphrase/joke: https://twitter.com/econbartleby/status ... 2157296643
molto tricky

User avatar
discovolante
After Pie
Posts: 2053
Joined: Fri Oct 11, 2019 5:10 pm

Re: mess

Post by discovolante » Fri Aug 14, 2020 10:13 am

shpalman wrote:
Fri Aug 14, 2020 10:01 am
https://www.thepoke.co.uk/2020/08/13/ga ... same-joke/

Gavin Williamson is quoted as saying that boosting exam grades 'would harm Generation Covid for life' while the words being attributed to him are a paraphrase/joke: https://twitter.com/econbartleby/status ... 2157296643
Boring.
don't get any big ideas, they're not gonna happen

User avatar
Little waster
Dorkwood
Posts: 1027
Joined: Tue Nov 12, 2019 12:35 am
Location: About 1 inch behind my eyes

Re: mess

Post by Little waster » Fri Aug 14, 2020 10:15 am

shpalman wrote:
Fri Aug 14, 2020 10:01 am
https://www.thepoke.co.uk/2020/08/13/ga ... same-joke/

Gavin Williamson is quoted as saying that boosting exam grades 'would harm Generation Covid for life' while the words being attributed to him are a paraphrase/joke: https://twitter.com/econbartleby/status ... 2157296643
That everyone just took it at face value probably gives too much insight into how the last 4 years of Trump, Brexit then COVID has traumatised our expectations. Even our joke detectors have developed 1000-yard stares. :|
People who use Godwin's Law are worse than Hitler.

User avatar
discovolante
After Pie
Posts: 2053
Joined: Fri Oct 11, 2019 5:10 pm

Re: mess

Post by discovolante » Fri Aug 14, 2020 10:47 am

Little waster wrote:
Fri Aug 14, 2020 10:15 am
shpalman wrote:
Fri Aug 14, 2020 10:01 am
https://www.thepoke.co.uk/2020/08/13/ga ... same-joke/

Gavin Williamson is quoted as saying that boosting exam grades 'would harm Generation Covid for life' while the words being attributed to him are a paraphrase/joke: https://twitter.com/econbartleby/status ... 2157296643
That everyone just took it at face value probably gives too much insight into how the last 4 years of Trump, Brexit then COVID has traumatised our expectations. Even our joke detectors have developed 1000-yard stares. :|
It's really not beyond the realms of plausibility.

Anyway sorry for the derail from sensible stuff.
don't get any big ideas, they're not gonna happen

FlammableFlower
Catbabel
Posts: 783
Joined: Mon Nov 11, 2019 1:22 pm

Re: mess

Post by FlammableFlower » Fri Aug 14, 2020 10:55 am

It was all so predictable...


...in fact so much so the person actually presented their written evidence to a parliamentary committee last month... and then has been shown to have been right.

User avatar
JQH
Dorkwood
Posts: 1010
Joined: Mon Nov 11, 2019 3:30 pm
Location: Sar Flandan

Re: mess

Post by JQH » Fri Aug 14, 2020 3:41 pm

discovolante wrote:
Fri Aug 14, 2020 10:13 am
shpalman wrote:
Fri Aug 14, 2020 10:01 am
https://www.thepoke.co.uk/2020/08/13/ga ... same-joke/

Gavin Williamson is quoted as saying that boosting exam grades 'would harm Generation Covid for life' while the words being attributed to him are a paraphrase/joke: https://twitter.com/econbartleby/status ... 2157296643
Boring.
And too late. The truth has not got its boots on but the joke is half way round the world.

Oh dear.
How sad.
Never mind.
And remember that if you botch the exit, the carnival of reaction may be coming to a town near you.

Fintan O'Toole

User avatar
Grumble
Dorkwood
Posts: 1291
Joined: Mon Nov 11, 2019 1:03 pm

Re: mess

Post by Grumble » Fri Aug 14, 2020 6:46 pm

Janey Godley wrote:ABBA are disgusted to find out they are now called DGAF #examshambles
I know this is vitriol, no solution, spleen venting, but I feel better having screamed, don’t you?

FlammableFlower
Catbabel
Posts: 783
Joined: Mon Nov 11, 2019 1:22 pm

Re: mess

Post by FlammableFlower » Fri Aug 14, 2020 9:19 pm

This is going to get interesting... The gov reckons everything is going to be fine re appeals, however lots of institutions are already full (mine certainly is) and won't be going into clearing. Anyone who didn't get in but successfully appeals and could have will be given an guaranteed place for next year. We simply can't take any more.

User avatar
Bird on a Fire
Light of Blast
Posts: 4243
Joined: Fri Oct 11, 2019 5:05 pm
Location: nadir of brie

Re: mess

Post by Bird on a Fire » Sat Aug 15, 2020 2:16 pm

What a travesty.

I hope universities will be making their displeasure extremely clear to Ofqual et al. - there's not much point in universities' efforts to widen participation if the government's going to penalise students for coming from a traditionally low-achieving area.
THINK BIG AND UPEND THE SYSTEM

#ShowYourStripes

User avatar
snoozeofreason
Clardic Fug
Posts: 181
Joined: Fri Nov 15, 2019 1:22 pm

Re: mess

Post by snoozeofreason » Sat Aug 15, 2020 6:45 pm

Dan Davies in the Guardian argues (IMO quite persuasively) that
The problem was fundamentally insoluble, from a mathematical point of view. If the system is dependent on exams to allocate the grades, but it can’t have the exams, then it can’t allocate the grades. No statistical method in the world is going to be able to give you good results if the information you’re looking for is fundamentally not there in the dataset that you’re trying to extract it from. (Hollywood is just wrong on this one when it has people looking at grainy CCTV footage and saying “Let’s enhance.”)
And I can well believe that
But even if, impossibly, all the problems had been solved, the Ofqual report on its own methodology gives the game away with respect to a much more serious issue. If there had been a perfect solution to the problem of a pandemic-hit examination process, so that every candidate was given exactly the grade that they would have got in an exam, how fair would this be? Turn to page 81 to see the answer and weep. In any subject other than maths, physics, chemistry, biology or psychology, there’s no better than a 70% chance that two markers of the same paper would agree on the grade to be assigned. Anecdotally, by the way, “two markers” in this context might mean “the same marker, at two different times”.
In six days the Lord made the heavens and the earth, the sea, and all that is in them. The human body was knocked up pretty late on the Friday afternoon, with a deadline looming. How well do you expect it to work?

Allo V Psycho
Clardic Fug
Posts: 219
Joined: Sat Nov 16, 2019 8:18 am

Re: mess

Post by Allo V Psycho » Sat Aug 15, 2020 7:30 pm

There is a fairly common psychometric problem at the root of this: that of the distinction between populations and individuals.

Predicted A level grades have always significantly over-predicted achieved grades. This may represent teacher/school optimism but also 'gaming' the system by selective/private schools. If you over-predict your students performance, then they are more likely to get an interview, especially for high competition subjects like medicine and dentistry, where AAA is the minimum. If/when they fail to achieve these inflated grades, then they still have the possibility of getting in as a near miss, if the course needs to match its desired numbers. A comprehensive school which forecasts its pupils' outcomes as accurately as possible, may also ensure that their pupils don't get the interview in the first place, but unfair advantage is why parents pay for their kids to go to selective/private schools.

So: the original COVID system was designed to reduce the population of predicted grades to the distribution of previous actual grades. In that it is quite successful. The problem is that this population strategy then impacts on individuals. So, for instance, matching the historical proportion of grades in a comprehensive school to the current predictions, does not work for the unusually gifted individual in a generally poorly performing school - which is the common situation in many comprehensives.

It is a similar problem to that of admissions. The population of students from selective/private schools has an unfair (i.e. not sustained by subsequent University performance) of 2 A level grades. So I could in theory subtract 2 A level grades from all selective/private school applicants to medicine. But I can't say that this is true of the individual applicant - Jemima for the private school might deserve her three As - and applications are made by individuals, not populations. SO if I turn Jemima down, I can be sued for discrimination.

So, for kids from comprehensive schools with a poor record, the population algorithm downgrades them as individuals, and they are rightly miffed, just like the example posted above in this thread.

User avatar
Gfamily
After Pie
Posts: 1858
Joined: Mon Nov 11, 2019 1:00 pm
Location: NW England

Re: mess

Post by Gfamily » Sat Aug 15, 2020 7:57 pm

Allo V Psycho wrote:
Sat Aug 15, 2020 7:30 pm
It is a similar problem to that of admissions. The population of students from selective/private schools has an unfair (i.e. not sustained by subsequent University performance) of 2 A level grades.
Is that unsustained two grade performance boost at A level a reliably confirmed figure?

It really fits my preconceptions about selective/private schools giving 'meh!' students a boost that can't be sustained through university, so it would be useful to know if it's reflected in real world data.
My avatar was a scientific result that was later found to be 'mistaken' - I rarely claim to be 100% correct
ETA 5/8/20: I've been advised that the result was correct, it was the initial interpretation that needed to be withdrawn
Meta? I'd say so!

User avatar
jimbob
After Pie
Posts: 1785
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: mess

Post by jimbob » Sat Aug 15, 2020 9:08 pm

Why wait until now for the results?


Presumably OFQUAL knew that they would be applying an algorithm based on predicted grades?

Why not tell people 4 months ago, and give them more time to sort it out?
Have you considered stupidity as an explanation

FlammableFlower
Catbabel
Posts: 783
Joined: Mon Nov 11, 2019 1:22 pm

Re: mess

Post by FlammableFlower » Sat Aug 15, 2020 9:14 pm

jimbob wrote:
Sat Aug 15, 2020 9:08 pm
Why wait until now for the results?


Presumably OFQUAL knew that they would be applying an algorithm based on predicted grades?

Why not tell people 4 months ago, and give them more time to sort it out?
A bl..dy good question.

I'm trying to figure out just what Williamson has been doing for the last 4 months.

User avatar
Little waster
Dorkwood
Posts: 1027
Joined: Tue Nov 12, 2019 12:35 am
Location: About 1 inch behind my eyes

Re: mess

Post by Little waster » Sat Aug 15, 2020 9:27 pm

FlammableFlower wrote:
Sat Aug 15, 2020 9:14 pm
jimbob wrote:
Sat Aug 15, 2020 9:08 pm
Why wait until now for the results?


Presumably OFQUAL knew that they would be applying an algorithm based on predicted grades?

Why not tell people 4 months ago, and give them more time to sort it out?
A bl..dy good question.

I'm trying to figure out just what Williamson has been doing for the last 4 months.
He's been very busy ensuring all students were back in school before the Summer break... Oh wait...
People who use Godwin's Law are worse than Hitler.

Allo V Psycho
Clardic Fug
Posts: 219
Joined: Sat Nov 16, 2019 8:18 am

Re: mess

Post by Allo V Psycho » Sun Aug 16, 2020 7:20 am

Gfamily wrote:
Sat Aug 15, 2020 7:57 pm
Allo V Psycho wrote:
Sat Aug 15, 2020 7:30 pm
It is a similar problem to that of admissions. The population of students from selective/private schools has an unfair (i.e. not sustained by subsequent University performance) of 2 A level grades.
Is that unsustained two grade performance boost at A level a reliably confirmed figure?

It really fits my preconceptions about selective/private schools giving 'meh!' students a boost that can't be sustained through university, so it would be useful to know if it's reflected in real world data.
Yes. See for example this HEFCE update of the effect, first summarised in 2002-2003

https://dera.ioe.ac.uk/19811/1/HEFCE2014_03.pdf
State school students tend to do better in their degree studies than students from
independent schools with the same prior educational attainment
15. This difference is less marked in women, those with the highest A-level achievement, and
those who study at HEIs with high entry tariffs, but even in these categories it remains
statistically significant.
16. This improved performance is not affected by the type of state school. Students from
community schools, foundation schools, sixth form colleges and voluntary controlled or aided
schools all tend to do better than their independent school counterparts with the same prior
educational attainment.
Incidentally, further up thread, Sciolus said
The other interesting thing is how much is coming out that shows the whole exams system is utterly unfit for purpose. If exam grades bear no resemblance to either mock grades or teacher forecasts, why should we believe that the exams are the true measure?
The data shows that despite the challenges, A levels measure something and that something correlates with later performance in a number of ways. The effect is surprisingly linear, and is statistically significant. However, the effect size allows for plenty of other factors to be present.

User avatar
bob sterman
Fuzzable
Posts: 307
Joined: Mon Nov 11, 2019 10:25 pm

Re: mess

Post by bob sterman » Sun Aug 16, 2020 8:52 am

Something that is not getting discussed much - given that revising and preparing for exams helps pupils/students consolidate their knowledge and understanding - we now have a cohort who were never required to do this for their final A-level exams.

For those who are going straight into university - they are entering degree programmes having not fully consolidated their knowledge and understanding of the material on their A-level syllabi.

So at a university - if we admit a bunch of students who have been awarded BBB in their A-levels - perhaps they all would have got BBB had they completed their exams. But it's likely they will arrive on their degree programmes far less well-prepared than students who actually completed their A-level exam preparation.

User avatar
jimbob
After Pie
Posts: 1785
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: mess

Post by jimbob » Sun Aug 16, 2020 9:40 am

Allo V Psycho wrote:
Sat Aug 15, 2020 7:30 pm
There is a fairly common psychometric problem at the root of this: that of the distinction between populations and individuals.

Predicted A level grades have always significantly over-predicted achieved grades. This may represent teacher/school optimism but also 'gaming' the system by selective/private schools. If you over-predict your students performance, then they are more likely to get an interview, especially for high competition subjects like medicine and dentistry, where AAA is the minimum. If/when they fail to achieve these inflated grades, then they still have the possibility of getting in as a near miss, if the course needs to match its desired numbers. A comprehensive school which forecasts its pupils' outcomes as accurately as possible, may also ensure that their pupils don't get the interview in the first place, but unfair advantage is why parents pay for their kids to go to selective/private schools.

So: the original COVID system was designed to reduce the population of predicted grades to the distribution of previous actual grades. In that it is quite successful. The problem is that this population strategy then impacts on individuals. So, for instance, matching the historical proportion of grades in a comprehensive school to the current predictions, does not work for the unusually gifted individual in a generally poorly performing school - which is the common situation in many comprehensives.

It is a similar problem to that of admissions. The population of students from selective/private schools has an unfair (i.e. not sustained by subsequent University performance) of 2 A level grades. So I could in theory subtract 2 A level grades from all selective/private school applicants to medicine. But I can't say that this is true of the individual applicant - Jemima for the private school might deserve her three As - and applications are made by individuals, not populations. SO if I turn Jemima down, I can be sued for discrimination.

So, for kids from comprehensive schools with a poor record, the population algorithm downgrades them as individuals, and they are rightly miffed, just like the example posted above in this thread.
The first phrase is key to a lot of things. The "scientific racists" and sexists fail to understand that. It also fits with Cummings's reported views when he was at university about state-school students being thick.

There is the nice bit about the algorithm being secret and failing to accept help from statisticians who didn't sign an NDA
Have you considered stupidity as an explanation

User avatar
Gfamily
After Pie
Posts: 1858
Joined: Mon Nov 11, 2019 1:00 pm
Location: NW England

Re: mess

Post by Gfamily » Sun Aug 16, 2020 9:42 am

Allo V Psycho wrote:
Sun Aug 16, 2020 7:20 am
Yes.
Thanks.
My avatar was a scientific result that was later found to be 'mistaken' - I rarely claim to be 100% correct
ETA 5/8/20: I've been advised that the result was correct, it was the initial interpretation that needed to be withdrawn
Meta? I'd say so!

User avatar
jimbob
After Pie
Posts: 1785
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: mess

Post by jimbob » Sun Aug 16, 2020 9:54 am

https://twitter.com/david_colquhoun/sta ... 0930908160
Replying to
@ParkinJim

@GavinWilliamson
and
@ofqual
I've been told that no ties were allowed. That's unbelievably silly, IMO
Have you considered stupidity as an explanation

User avatar
jimbob
After Pie
Posts: 1785
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: mess

Post by jimbob » Sun Aug 16, 2020 9:57 am

Also the the sudden scrapping of Public Health England.
Have you considered stupidity as an explanation

FlammableFlower
Catbabel
Posts: 783
Joined: Mon Nov 11, 2019 1:22 pm

Re: mess

Post by FlammableFlower » Sun Aug 16, 2020 10:25 am

Looks like they've had a bit of a panic after their initial criteria for appeals came out and was thoroughly denounced and contradicted the minister's comments regarding "safety net" provisions and are going away to think about it...

Again.... 4 months guys, you had 4 months.

User avatar
mediocrity511
Fuzzable
Posts: 288
Joined: Mon Nov 11, 2019 2:16 pm

Re: mess

Post by mediocrity511 » Sun Aug 16, 2020 1:13 pm

Allo V Psycho wrote:
Sat Aug 15, 2020 7:30 pm
There is a fairly common psychometric problem at the root of this: that of the distinction between populations and individuals.

Predicted A level grades have always significantly over-predicted achieved grades. This may represent teacher/school optimism but also 'gaming' the system by selective/private schools. If you over-predict your students performance, then they are more likely to get an interview, especially for high competition subjects like medicine and dentistry, where AAA is the minimum. If/when they fail to achieve these inflated grades, then they still have the possibility of getting in as a near miss, if the course needs to match its desired numbers. A comprehensive school which forecasts its pupils' outcomes as accurately as possible, may also ensure that their pupils don't get the interview in the first place, but unfair advantage is why parents pay for their kids to go to selective/private schools.

So: the original COVID system was designed to reduce the population of predicted grades to the distribution of previous actual grades. In that it is quite successful. The problem is that this population strategy then impacts on individuals. So, for instance, matching the historical proportion of grades in a comprehensive school to the current predictions, does not work for the unusually gifted individual in a generally poorly performing school - which is the common situation in many comprehensives.

It is a similar problem to that of admissions. The population of students from selective/private schools has an unfair (i.e. not sustained by subsequent University performance) of 2 A level grades. So I could in theory subtract 2 A level grades from all selective/private school applicants to medicine. But I can't say that this is true of the individual applicant - Jemima for the private school might deserve her three As - and applications are made by individuals, not populations. SO if I turn Jemima down, I can be sued for discrimination.

So, for kids from comprehensive schools with a poor record, the population algorithm downgrades them as individuals, and they are rightly miffed, just like the example posted above in this thread.
One minor niggle about what you've said here, although I agree with much of it. The centre assessed grades that teachers submitted for these A level results are a very different thing from UCAS grades. Teachers know that UCAS offers are a bit like a bid for a university place, take place at the start of year 13 and are invariably optimistic. What they submitted for the A level results weren't UCAS predictions, they were much more realistic and evidence based and they took into account previous achievement data for both the school and individual pupils. They were warned their submissions would be compared to historical attainment data. So when these more realistic grades were downgraded, they will fairly frequently have already been a downgrade from what pupils received as a UCAS estimate.

User avatar
Gfamily
After Pie
Posts: 1858
Joined: Mon Nov 11, 2019 1:00 pm
Location: NW England

Re: mess

Post by Gfamily » Sun Aug 16, 2020 2:26 pm

Shared from someone else
‘Ah Gavin, please sit down. I’m going to begin your appraisal by looking back over the last few years.’
‘Few years? I’ve only been doing the job for a year.’
‘Yes, but we need to look at your predecessors.’
‘Why? That doesn’t affect how I do my job.’
‘I’m afraid it does. Let’s start with Damian. He didn’t do much to be honest. Basically kept the seat warm for you. Now Justine. We had very high hopes for her. Lots of enthusiasm but she was too friendly with the teachers. Sadly, she had to go. What to say about Nicky? Made the mistake of thinking she could do the job in the same way as the previous minister. No originality. Something you can’t accuse Michael of. Totally overhauled the system. Of course, it did annoy quite a few people - teachers, unions, parents and pupils. Quite a lot to live up to there, eh Gavin?’
‘Yes but I think I’ve managed the last one.’
‘Annoy teachers, unions, parents and pupils? Yes, I think you’ve managed that. But overall, based on all your predecessors, I’m going to have to mark you down.’
‘But that’s not fair? What about the work I’ve done this year?’
‘Doesn’t count for much. You see we have to base your appraisal on how others have done the job.’
‘That makes no sense whatsoever.’
‘No, it doesn’t. Now you know how the years’ exam students feel.’
My avatar was a scientific result that was later found to be 'mistaken' - I rarely claim to be 100% correct
ETA 5/8/20: I've been advised that the result was correct, it was the initial interpretation that needed to be withdrawn
Meta? I'd say so!

raven
Fuzzable
Posts: 244
Joined: Mon Nov 11, 2019 8:58 pm

Re: mess

Post by raven » Sun Aug 16, 2020 3:08 pm

mediocrity511 wrote:
Sun Aug 16, 2020 1:13 pm
Allo V Psycho wrote:
Sat Aug 15, 2020 7:30 pm
There is a fairly common psychometric problem at the root of this: that of the distinction between populations and individuals.

[...]

So: the original COVID system was designed to reduce the population of predicted grades to the distribution of previous actual grades. In that it is quite successful. The problem is that this population strategy then impacts on individuals. So, for instance, matching the historical proportion of grades in a comprehensive school to the current predictions, does not work for the unusually gifted individual in a generally poorly performing school - which is the common situation in many comprehensives.
What they submitted for the A level results weren't UCAS predictions, they were much more realistic and evidence based and they took into account previous achievement data for both the school and individual pupils.
My bold.

I reckon a lot of the apparent unfairness comes down to applying whole-school statistics to the individual.

But that's how we've been handling education data for a while. Back when mine were entering secondary over 10 years ago now, the system took their KeyStage 2 results, combined those with school-level 'value-added' data -- ie, an estimate of how pupils usually progressed at that particular school - and used that to produce individual targets for GCSE. Sometimes those were aspirational targets, based on 'if this was a school in the top 25% of schools' & got more progress, which was reassuring because I always felt that setting targets that way might create a self-limiting effect of 'well, pupils like you at this school don't get higher grades than this'. But that was how it worked, by applying whole-school data to individuals, probably because successive governments had been focused on monitoring school performance and improvement and predicting individual targets/grades was just part and parcel of that, and seemed to become a stick to beat teachers with - get these results or we'll want to know why not, kind of thing.

And, iirc, the estimated targets from that were notoriously unreliable for anything outside English, Maths, and maybe Science. (When youngest, who was a bright but lazy and not very co-ordinated or musical child, brought home A* targets in Art, PE & Music, we just laughed and rolled our eyes.)

Teacher estimates were much more realistic though.

Post Reply