Incomes, education and inequality

Discussions about serious topics, for serious people
monkey
After Pie
Posts: 1906
Joined: Wed Nov 13, 2019 5:10 pm

Re: Incomes, education and inequality

Post by monkey » Wed Apr 12, 2023 5:02 pm

dyqik wrote:
Wed Apr 12, 2023 12:28 am
Getting in is something that needs considerable investment [in] schools, of course.
This is solely what my post was commenting on.

I know scholarships exist, but they are worth f.ck all if you can't get the grades because of extra hurdles to jump.



(Also, I'm also assuming that you're using the USian version of middle class, because we're talking about the US)

User avatar
Sciolus
Dorkwood
Posts: 1314
Joined: Mon Nov 11, 2019 6:42 pm

Re: Incomes, education and inequality

Post by Sciolus » Wed Apr 12, 2023 9:17 pm

Let's cast our minds back a year or two, when Ofqual made it absolutely clear that a large part of exam results -- a couple of grades' worth -- is down to which school you go to, and that this, far from being a bad thing that they try to compensate for, is actually an intentional, deliberate and valued feature of the system that they went out of their way to reproduce.

User avatar
dyqik
Princess POW
Posts: 7527
Joined: Wed Sep 25, 2019 4:19 pm
Location: Masshole
Contact:

Re: Incomes, education and inequality

Post by dyqik » Thu Apr 13, 2023 1:23 am

monkey wrote:
Wed Apr 12, 2023 5:02 pm
dyqik wrote:
Wed Apr 12, 2023 12:28 am
Getting in is something that needs considerable investment [in] schools, of course.
This is solely what my post was commenting on.

I know scholarships exist, but they are worth f.ck all if you can't get the grades because of extra hurdles to jump.



(Also, I'm also assuming that you're using the USian version of middle class, because we're talking about the US)
Without national exams, grades are a bit complicated in the US (AIUI, I have not figured out how a school GPA turns into something that can be compared between schools). It's doing the work to put together a proper application package, with the appropriate extracurricular elements and AP courses etc. that makes the most difference.

For practical purposes here, I'm defining middle class as "has some resources to dedicate to children's education outside of mandatory education, but not the kind of money that pays for private education".

User avatar
Formerly AvP
Stargoon
Posts: 119
Joined: Sun Oct 16, 2022 6:42 pm

Re: Incomes, education and inequality

Post by Formerly AvP » Thu Apr 13, 2023 5:25 am

Sciolus wrote:
Wed Apr 12, 2023 9:17 pm
Let's cast our minds back a year or two, when Ofqual made it absolutely clear that a large part of exam results -- a couple of grades' worth -- is down to which school you go to, and that this, far from being a bad thing that they try to compensate for, is actually an intentional, deliberate and valued feature of the system that they went out of their way to reproduce.
Hi Sciolus, can I ask what this refers to?
Was Allo V Psycho, but when my laptop died, I lost all the info on it...

User avatar
jimbob
Light of Blast
Posts: 5276
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: Incomes, education and inequality

Post by jimbob » Thu Apr 13, 2023 7:22 am

Formerly AvP wrote:
Thu Apr 13, 2023 5:25 am
Sciolus wrote:
Wed Apr 12, 2023 9:17 pm
Let's cast our minds back a year or two, when Ofqual made it absolutely clear that a large part of exam results -- a couple of grades' worth -- is down to which school you go to, and that this, far from being a bad thing that they try to compensate for, is actually an intentional, deliberate and valued feature of the system that they went out of their way to reproduce.
Hi Sciolus, can I ask what this refers to?
The infamous algorithm thought up to assign A level and then GCSE grades

https://blogs.lse.ac.uk/impactofsocials ... ng-fiasco/

In the absence of actual exams, Ofqual decided to estimate the A-level grades using an algorithm. Three inputs were used: 1. The historical grade distribution of schools from the three previous years (2017-2019); 2. The rank of each student within her own school for a particular subject, based on a teacher’s evaluation of their likely grade had the A-levels gone forward as planned (called the “Centre Assessed Grade” or CAG for short); 3. The previous exam results for a student per subject.

The algorithm looked at the historical grade distribution of a school and then decided a students’ grade on the basis of their ranking. For instance, if a student was halfway down the ranking list, then her grade would be roughly equal to what the person in the same ranking obtained in previous years. This approach was intended to correct for observed grade inflation in the CAGs (pdf), which explains why the algorithm’s grades were lower than the scores students expected. However, the use of historical data in algorithms is also a key component in latent algorithmic bias.
And this beauty






Several people identified issues with Ofqual’s algorithm from a technical report that was released by the UK government. Among other things, experts criticized the low accuracy of the algorithm and lack of uncertainty bounds for the resulting grades. Meanwhile, public outcry centred on the algorithm’s unfair results. For instance, if no one from your school has gotten the highest grade in the past three years, it’s extremely unlikely—if not impossible—for anyone from your school to attain that grade this year.

In addition, the algorithm puts more weight on the CAGs if there are fewer than 15 students in a particular subject at a particular school. That meant students at smaller schools were more likely to benefit from grade inflation than those at larger schools. This approach reinforces existing inequalities, as one analysis showed that the “proportion of A* and As awarded to independent (fee-paying) schools rose by 4.7 percentage points—more than double the rate for state comprehensive schools.”’
Have you considered stupidity as an explanation

User avatar
Sciolus
Dorkwood
Posts: 1314
Joined: Mon Nov 11, 2019 6:42 pm

Re: Incomes, education and inequality

Post by Sciolus » Thu Apr 13, 2023 7:38 am

Thanks jimbob.

User avatar
Formerly AvP
Stargoon
Posts: 119
Joined: Sun Oct 16, 2022 6:42 pm

Re: Incomes, education and inequality

Post by Formerly AvP » Thu Apr 13, 2023 5:01 pm

Yes, thanks jimbob too. Inappropriate logic was used in these algorithms.
Was Allo V Psycho, but when my laptop died, I lost all the info on it...

User avatar
Sciolus
Dorkwood
Posts: 1314
Joined: Mon Nov 11, 2019 6:42 pm

Re: Incomes, education and inequality

Post by Sciolus » Thu Apr 13, 2023 8:00 pm

No, the algorithm largely did what it was designed to do, which was to mimic pupils' exam results at a gross level. That's why it had to fudge their actual performance (CAGs) based on which school they went to. The scandal is that the exam system the algorithm was mimicking places such weight on which school you go to, instead of individual ability.

User avatar
jimbob
Light of Blast
Posts: 5276
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: Incomes, education and inequality

Post by jimbob » Thu Apr 13, 2023 8:25 pm

Sciolus wrote:
Thu Apr 13, 2023 8:00 pm
No, the algorithm largely did what it was designed to do, which was to mimic pupils' exam results at a gross level. That's why it had to fudge their actual performance (CAGs) based on which school they went to. The scandal is that the exam system the algorithm was mimicking places such weight on which school you go to, instead of individual ability.
It really was badly designed and saying that it met the specification is no excuse when the specification was messed up


From the thread on it.
Brightonian wrote:
Thu Aug 13, 2020 1:18 pm
Sky News wrote:Ofqual has warned that appealing against a grade will affect other students from the same school because of the rank order system.

It said if one student successfully appealed against their position in the rank order it would push another student down the ranking and they would then need to be allowed to appeal.
https://news.sky.com/story/as-and-a-lev ... d-12046680
FlammableFlower wrote:
Fri Aug 14, 2020 10:55 am
It was all so predictable...


...in fact so much so the person actually presented their written evidence to a parliamentary committee last month... and then has been shown to have been right.

jimbob wrote:
Sun Aug 16, 2020 9:54 am
https://twitter.com/david_colquhoun/sta ... 0930908160
Replying to
@ParkinJim

@GavinWilliamson
and
@ofqual
I've been told that no ties were allowed. That's unbelievably silly, IMO

jimbob wrote:
Sun Aug 16, 2020 9:15 pm
Gfamily wrote:
Sun Aug 16, 2020 8:58 pm
I'm sure I read somewhere that for some centres they didn't use the CAGs, just the laddering, and applied the historic distribution of grades.

But I'm not sure where, and I may have misunderstood what I read

Ah, it read in the Good Law Project article.
I saw that classes of more than 15 people would be calculated without any reference to teachers’ predicted grades, and I thought that wasn’t right,” he told the Guardian
I've seen it stated that there were different rules applied to large and small classes.

Has the algorithm been released?

https://twitter.com/david_colquhoun/sta ... 4558369792
David Colquhoun
@david_colquhoun
This is incredible -
"A-levels: Exam regulator ignored expert help after statisticians wouldn't sign non-disclosure agreements"
Orwellian
Quote Tweet
Alastair Horne
@pressfuturist
· Aug 14
Wow. Ofqual turned down the offer of advice from the Royal Statistical Society on its algorithm because the RSS refused to be silent about any flaws it might find. https://news.sky.com/story/a-levels-exa ... s-12049289
Show this thread
5:51 PM · Aug 14, 2020·TweetDeck
Have you considered stupidity as an explanation

User avatar
jimbob
Light of Blast
Posts: 5276
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: Incomes, education and inequality

Post by jimbob » Thu Apr 13, 2023 8:26 pm

And found a post where private schools managed to get their grades inflated.
jimbob wrote:
Mon Aug 17, 2020 10:42 am
https://twitter.com/queenofswords6/stat ... 85667?s=20
See new Tweets
Conversation
Dr Georgina Porter
@queenofswords6
·
Aug 15
A levels...short thread

If anyone doubts who the winners are this year, here's a random selection of independent schools and the % of A*-A grades this year.

2019 figures follow in brackets.

Mayfield Girls' 66% (55%)
Sherfield School 70% (37%)
Leweston School 51% (37%)
Dr Georgina Porter
@queenofswords6
Farlington School 62% (48%)
The Marist School 55% (52%)
Godolphin School 50% (29.2%)
St Mary's Cambridge 54% (51%)
St George's Ascot 63% (35%)
Rye St Anthony 48.1% (18.3%)

A shout out to the dunces at the last school on the list who said its A-A*s had 'gone up 30%' (it's +165%)
4:17 AM · Aug 15, 2020·Twitter Web App
Have you considered stupidity as an explanation

User avatar
Sciolus
Dorkwood
Posts: 1314
Joined: Mon Nov 11, 2019 6:42 pm

Re: Incomes, education and inequality

Post by Sciolus » Fri Apr 14, 2023 7:41 am

jimbob wrote:
Thu Apr 13, 2023 8:25 pm
Sciolus wrote:
Thu Apr 13, 2023 8:00 pm
No, the algorithm largely did what it was designed to do, which was to mimic pupils' exam results at a gross level. That's why it had to fudge their actual performance (CAGs) based on which school they went to. The scandal is that the exam system the algorithm was mimicking places such weight on which school you go to, instead of individual ability.
It really was badly designed and saying that it met the specification is no excuse when the specification was messed up
I don't think we are disagreeing. My point is that it's not just the algorithm but the system in which it sits that is messed up.

User avatar
jimbob
Light of Blast
Posts: 5276
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: Incomes, education and inequality

Post by jimbob » Fri Apr 14, 2023 8:28 am

Sciolus wrote:
Fri Apr 14, 2023 7:41 am
jimbob wrote:
Thu Apr 13, 2023 8:25 pm
Sciolus wrote:
Thu Apr 13, 2023 8:00 pm
No, the algorithm largely did what it was designed to do, which was to mimic pupils' exam results at a gross level. That's why it had to fudge their actual performance (CAGs) based on which school they went to. The scandal is that the exam system the algorithm was mimicking places such weight on which school you go to, instead of individual ability.
It really was badly designed and saying that it met the specification is no excuse when the specification was messed up
I don't think we are disagreeing. My point is that it's not just the algorithm but the system in which it sits that is messed up.
Ah, agreed
Have you considered stupidity as an explanation

User avatar
Formerly AvP
Stargoon
Posts: 119
Joined: Sun Oct 16, 2022 6:42 pm

Re: Incomes, education and inequality

Post by Formerly AvP » Fri Apr 14, 2023 11:13 am

jimbob wrote:
Fri Apr 14, 2023 8:28 am
Sciolus wrote:
Fri Apr 14, 2023 7:41 am
jimbob wrote:
Thu Apr 13, 2023 8:25 pm


It really was badly designed and saying that it met the specification is no excuse when the specification was messed up
I don't think we are disagreeing. My point is that it's not just the algorithm but the system in which it sits that is messed up.
Ah, agreed
I'm not sure I would describe the algorithm modellers as 'statistically illiterate'. The full Research and Analysis report
https://assets.publishing.service.gov.u ... report.pdf
addresses some reasonably sophisticated statistics (multi-level modelling gets a mention, for instance). What I suspect is that they were naive about the performance of actual students - they may have been good statisticians, but have little experience in how classes of real students are likely to perform.

For instance, it indeed appears to be the case that ties were not allowed in the rank order of students. Ties are a challenge in systems designed to rank (as I know only too well in my day job). It's mathematically convenient, neater, to just 'forbid' them in the model. But this has negative consequences. If it is true that a successful appeal would result in all students below the new level being downgraded by one rank position, then this makes no real world sense. Equally, matching the outcomes by school to previous years may make sense for the system as a whole, but does not make sense for individual unique school years.

A real teacher with a real class, however, may well be unable to distinguish the number of levels of performance required to rank uniquely all the students in that class. A unique ranking therefore has lower information content than a ranking with ties. And teachers also know that performance can vary quite markedly from year to year. If instead each student were assigned a score, that would both allow ties, and also allow a teacher to identify unusual excellent performance. So if in class where students had performed consistently at ‘B’ level, a particular candidate was unusually gifted, the CAG for that candidate could also be unusually high. The information content would therefore be higher than a ranking-based system. The algorithm could ensure that at a national level the outcomes were commensurate with previous years, but there would be a better match at the individual and school level. I suspect that what was needed in the algorithm design was not more expert statisticians, but more experienced and reflective teachers…

It reminds me of a problem with admissions to St George’s Medical School in the 1980s. A programmer was tasked to develop a computer programme which would match the hand-scoring of applications (ironically, to reduce inconsistencies). He analysed several years of data, and eventually came up with a computer model with a 90-95% match to historical data. St George’s were delighted and promptly employed it to select their students for interview.
You know where this is going…
A staff member with an interest in such things noticed that the programme applied a numerical penalty for being female and having a ‘funny name’ (ethnicity was inferred from surnames). Students were ranked, and those with high rankings were given an interview. Because it was written into the code, there could be no doubt about the presence of discrimination. The programmer was perhaps indignant – “You told me to model the existing system, and that is what I did!”
Was Allo V Psycho, but when my laptop died, I lost all the info on it...

User avatar
jimbob
Light of Blast
Posts: 5276
Joined: Mon Nov 11, 2019 4:04 pm
Location: High Peak/Manchester

Re: Incomes, education and inequality

Post by jimbob » Fri Apr 14, 2023 11:27 am

Formerly AvP wrote:
Fri Apr 14, 2023 11:13 am
jimbob wrote:
Fri Apr 14, 2023 8:28 am
Sciolus wrote:
Fri Apr 14, 2023 7:41 am

I don't think we are disagreeing. My point is that it's not just the algorithm but the system in which it sits that is messed up.
Ah, agreed
I'm not sure I would describe the algorithm modellers as 'statistically illiterate'. The full Research and Analysis report
https://assets.publishing.service.gov.u ... report.pdf
addresses some reasonably sophisticated statistics (multi-level modelling gets a mention, for instance). What I suspect is that they were naive about the performance of actual students - they may have been good statisticians, but have little experience in how classes of real students are likely to perform.

For instance, it indeed appears to be the case that ties were not allowed in the rank order of students. Ties are a challenge in systems designed to rank (as I know only too well in my day job). It's mathematically convenient, neater, to just 'forbid' them in the model. But this has negative consequences. If it is true that a successful appeal would result in all students below the new level being downgraded by one rank position, then this makes no real world sense. Equally, matching the outcomes by school to previous years may make sense for the system as a whole, but does not make sense for individual unique school years.

A real teacher with a real class, however, may well be unable to distinguish the number of levels of performance required to rank uniquely all the students in that class. A unique ranking therefore has lower information content than a ranking with ties. And teachers also know that performance can vary quite markedly from year to year. If instead each student were assigned a score, that would both allow ties, and also allow a teacher to identify unusual excellent performance. So if in class where students had performed consistently at ‘B’ level, a particular candidate was unusually gifted, the CAG for that candidate could also be unusually high. The information content would therefore be higher than a ranking-based system. The algorithm could ensure that at a national level the outcomes were commensurate with previous years, but there would be a better match at the individual and school level. I suspect that what was needed in the algorithm design was not more expert statisticians, but more experienced and reflective teachers…

It reminds me of a problem with admissions to St George’s Medical School in the 1980s. A programmer was tasked to develop a computer programme which would match the hand-scoring of applications (ironically, to reduce inconsistencies). He analysed several years of data, and eventually came up with a computer model with a 90-95% match to historical data. St George’s were delighted and promptly employed it to select their students for interview.
You know where this is going…
A staff member with an interest in such things noticed that the programme applied a numerical penalty for being female and having a ‘funny name’ (ethnicity was inferred from surnames). Students were ranked, and those with high rankings were given an interview. Because it was written into the code, there could be no doubt about the presence of discrimination. The programmer was perhaps indignant – “You told me to model the existing system, and that is what I did!”

Maybe, but equally, if one designs an algorithm that has a discontinuity between classes of 15 and 16 kids, that's bad
Have you considered stupidity as an explanation

Post Reply