has education funding increased?

Discussions about serious topics, for serious people
sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: has education funding increased?

Post by sheldrake » Mon Jan 06, 2020 12:18 pm

plodder wrote:
Mon Jan 06, 2020 12:12 pm
I can guess. Perhaps AvP is simply trying to understand what you're actually trying to say based on what you've written, which is how internet forums work.
He seems to be meandering about ignoring the research in peer-reviewed journals which I've posted, mistakenly attributing it as my own personal opinion and then getting confused about a post from days ago about 'fairness'.

Education funding has increased per pupil and A-levels got easier. We've established that at least so far. The only interesting outstanding question as far as I'm concerned in this thread is 'can you show that the easier A-levels actually correlates with lower standards'

Allo V Psycho
Catbabel
Posts: 735
Joined: Sat Nov 16, 2019 8:18 am

Re: has education funding increased?

Post by Allo V Psycho » Mon Jan 06, 2020 2:26 pm

sheldrake wrote:
Mon Jan 06, 2020 12:18 pm
plodder wrote:
Mon Jan 06, 2020 12:12 pm
I can guess. Perhaps AvP is simply trying to understand what you're actually trying to say based on what you've written, which is how internet forums work.
He seems to be meandering about ignoring the research in peer-reviewed journals which I've posted, mistakenly attributing it as my own personal opinion and then getting confused about a post from days ago about 'fairness'.

Education funding has increased per pupil and A-levels got easier. We've established that at least so far. The only interesting outstanding question as far as I'm concerned in this thread is 'can you show that the easier A-levels actually correlates with lower standards'
:-)

Are you struggling here, Sheldrake?
He seems to be meandering about ignoring the research in peer-reviewed journals which I've posted
None of your links are to peer reviewed journals. Only one of your popular sources refers to a peer reviewed journal, and the article is technically unconvincing. You have not sustained the argument you wish to support. Since I listed and discussed all of your links, I don't think it can really be said that I ignored them.
mistakenly attributing it as my own personal opinion
I do indeed think it is your personal opinion that standards have fallen. Do you wish to recede from this position?
then getting confused about a post from days ago about 'fairness
Someone is certainly confused. You asked about fairness in A levels, and then asserted you had not raised the issue of fairness. When I pointed out where you had done so, you then proposed that you were referring to my definition of fairness, when I hadn't in fact provided a definition of fairness.

Whether or not this is clear, I leave, with a smile, to the judgement of others.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: has education funding increased?

Post by sheldrake » Mon Jan 06, 2020 2:29 pm

I asked about fairness in the context of somebody else who was concerned about fairness in medical admissions. I did not raise the issue and it isn't one of my concerns. I do not think the research counts as my own opinion, because I'm not the one who published it.

If you think the research is unconvincing, explain why

plodder
Stummy Beige
Posts: 2981
Joined: Mon Nov 11, 2019 1:50 pm

Re: has education funding increased?

Post by plodder » Mon Jan 06, 2020 2:36 pm

Allo has already quoted the authors of the one report you've referenced, which you've not responded to - you were invited to explain why you found it compelling.

The authors said:
“there is no objective way in which comparisons over time are possible (Cresswell, 1996). Therefore all statements of changes in examination performance must be seen through the lens of contemporary, if expert, value judgements”.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: has education funding increased?

Post by sheldrake » Mon Jan 06, 2020 2:48 pm

plodder wrote:
Mon Jan 06, 2020 2:36 pm
Allo has already quoted the authors of the one report you've referenced, which you've not responded to - you were invited to explain why you found it compelling.

The authors said:
“there is no objective way in which comparisons over time are possible (Cresswell, 1996). Therefore all statements of changes in examination performance must be seen through the lens of contemporary, if expert, value judgements”.
The authors aren't arguing that there wasn't a decline in standards by saying that, they're saying that they had to have experts examine the papers to determine that.

Allo V Psycho
Catbabel
Posts: 735
Joined: Sat Nov 16, 2019 8:18 am

Re: has education funding increased?

Post by Allo V Psycho » Mon Jan 06, 2020 3:13 pm

sheldrake wrote:
Mon Jan 06, 2020 2:29 pm
I asked about fairness in the context of somebody else who was concerned about fairness in medical admissions. I did not raise the issue and it isn't one of my concerns. I do not think the research counts as my own opinion, because I'm not the one who published it.

If you think the research is unconvincing, explain why
I think you will find that I did, as Plodder is trying to indicate. The study was small, with only a few time points, and the scoring was done by Maths graduate students, not anyone with school level assessment technical expertise, and the authors themselves summarised the limitations.

I could have added that it was confined to one subject, that the comparative judgement methodology is not the best available, and that it did not explore the relevant curricula at the time of administration. I didn't want to be over-detailed: but am happy to do so now.

I don't blame the authors. They could only get access to a few papers, and the scorers were plainly the only cohort the authors could round up. That is why they were properly relatively tentative in their conclusions, and, again quite properly, list the limitations themselves.

This kind of nuance tends not to make it into press releases and newspaper reports, especially when those reporting may have an agenda of their own. This is why I asked you what, in your reading of the original research paper, made it convincing. I do take it you have read the original research paper.

Incidentally, I would say that you did indeed raise the issue of fairness directly with me, as shown in the quote I gave earlier, but, hey, I don't feel very strongly about your retrospective interpretation of semantics. Others can decide if it is reasonable or not.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: has education funding increased?

Post by sheldrake » Mon Jan 06, 2020 3:20 pm

The quote you gave earlier was a response to somebody who felt using A-level grades as medschool entrance criteria led to unfair results. It was not about me suggesting something was unfair.

The authors' summary of the limitations did not compromise the conclusion quoted.

plodder
Stummy Beige
Posts: 2981
Joined: Mon Nov 11, 2019 1:50 pm

Re: has education funding increased?

Post by plodder » Mon Jan 06, 2020 4:00 pm

you’re about to tell us why.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: has education funding increased?

Post by sheldrake » Mon Jan 06, 2020 4:19 pm

plodder wrote:
Mon Jan 06, 2020 4:00 pm
you’re about to tell us why.
Because they didn't say they thought their analysis was wrong, just that it used expert assessment of the papers rather than something purely numeric.

Allo V Psycho
Catbabel
Posts: 735
Joined: Sat Nov 16, 2019 8:18 am

Re: has education funding increased?

Post by Allo V Psycho » Mon Jan 06, 2020 9:07 pm

sheldrake wrote:
Mon Jan 06, 2020 4:19 pm
plodder wrote:
Mon Jan 06, 2020 4:00 pm
you’re about to tell us why.
Because they didn't say they thought their analysis was wrong, just that it used expert assessment of the papers rather than something purely numeric.
Ha,ha! Nope.
This is quite good fun.

That#s actually one of the weal points. Their 'experts' were recent maths graduates. As such, they were familiar with recent style questions, from current syllabi. these would seem easy. Questions from different historical syllabi would automatically seem harder.

There is an obvious question that Sheldrake hasn't asked me. If anyone wants to DM me any version of it, I'll give £20 to the charity of their choice. This includes Sheldrake.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: has education funding increased?

Post by sheldrake » Mon Jan 06, 2020 9:42 pm

Allo V Psycho wrote:
Mon Jan 06, 2020 9:07 pm

That#s actually one of the weal points. Their 'experts' were recent maths graduates. As such, they were familiar with recent style questions, from current syllabi. these would seem easy. Questions from different historical syllabi would automatically seem harder.
No they wouldn't. Older maths graduates I know also think newer questions are easier. This point doesn't make sense at all.

Allo V Psycho
Catbabel
Posts: 735
Joined: Sat Nov 16, 2019 8:18 am

Re: has education funding increased?

Post by Allo V Psycho » Tue Jan 07, 2020 10:19 am

sheldrake wrote:
Mon Jan 06, 2020 9:42 pm
Allo V Psycho wrote:
Mon Jan 06, 2020 9:07 pm

That#s actually one of the weal points. Their 'experts' were recent maths graduates. As such, they were familiar with recent style questions, from current syllabi. these would seem easy. Questions from different historical syllabi would automatically seem harder.
No they wouldn't. Older maths graduates I know also think newer questions are easier. This point doesn't make sense at all.
(sigh) But your 'older maths graduates' who think this are an anecdote of yours on the internet, as is their number, expertise and indeed existence, not authors in the paper, and therefore do not strengthen the paper itself, which is what we are discussing.

Sometimes, Shelley, I wonder if you argue in good faith...

cvb
Clardic Fug
Posts: 228
Joined: Tue Nov 12, 2019 8:15 am

Re: has education funding increased?

Post by cvb » Tue Jan 07, 2020 11:58 am

Allo V Psycho wrote:
Tue Jan 07, 2020 10:19 am
sheldrake wrote:
Mon Jan 06, 2020 9:42 pm
Allo V Psycho wrote:
Mon Jan 06, 2020 9:07 pm

That#s actually one of the weal points. Their 'experts' were recent maths graduates. As such, they were familiar with recent style questions, from current syllabi. these would seem easy. Questions from different historical syllabi would automatically seem harder.
No they wouldn't. Older maths graduates I know also think newer questions are easier. This point doesn't make sense at all.
(sigh) But your 'older maths graduates' who think this are an anecdote of yours on the internet, as is their number, expertise and indeed existence, not authors in the paper, and therefore do not strengthen the paper itself, which is what we are discussing.

Sometimes, Shelley, I wonder if you argue in good faith...
Only sometimes?

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: has education funding increased?

Post by sheldrake » Tue Jan 07, 2020 1:30 pm

Allo V Psycho wrote:
Tue Jan 07, 2020 10:19 am


(sigh) But your 'older maths graduates' who think this are an anecdote of yours on the internet, as is their number, expertise and indeed existence, not authors in the paper, and therefore do not strengthen the paper itself, which is what we are discussing.

Sometimes, Shelley, I wonder if you argue in good faith...
You're making an assumption that the age of the maths graduates is affecting the result in one direction where I don't think it is warranted.

plodder
Stummy Beige
Posts: 2981
Joined: Mon Nov 11, 2019 1:50 pm

Re: has education funding increased?

Post by plodder » Tue Jan 07, 2020 2:56 pm

You're proposing a piece of academic research as evidence for your argument, then bolting bits of anecdote, hunch and supposition onto the research when it turns out that the academics don't say what you want them to.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: has education funding increased?

Post by sheldrake » Tue Jan 07, 2020 2:57 pm

plodder wrote:
Tue Jan 07, 2020 2:56 pm
You're proposing a piece of academic research as evidence for your argument, then bolting bits of anecdote, hunch and supposition onto the research when it turns out that the academics don't say what you want them to.
As an answer to somebody trying to dismiss the research with their own conjecture.

plodder
Stummy Beige
Posts: 2981
Joined: Mon Nov 11, 2019 1:50 pm

Re: has education funding increased?

Post by plodder » Tue Jan 07, 2020 3:13 pm

No, that's the authors themselves who are saying that longitudinal effects can't be easily unpicked. Allo has just quoted them and agreed with them.

And even if someone was trying to dismiss it using conjecture (which they aren't), it doesn't mean you can stick some anecdote onto some research and say that your anecdote has the same validity.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: has education funding increased?

Post by sheldrake » Tue Jan 07, 2020 3:37 pm

plodder wrote:
Tue Jan 07, 2020 3:13 pm
No, that's the authors themselves who are saying that longitudinal effects can't be easily unpicked. Allo has just quoted them and agreed with them.

And even if someone was trying to dismiss it using conjecture (which they aren't), it doesn't mean you can stick some anecdote onto some research and say that your anecdote has the same validity.
They did not agree that the longitudinal effects all point in one direction, that is Allo's conjecture.

plodder
Stummy Beige
Posts: 2981
Joined: Mon Nov 11, 2019 1:50 pm

Re: has education funding increased?

Post by plodder » Tue Jan 07, 2020 6:17 pm

No, it's yours. Allo is quoting the authors as saying you can't draw any reliable results either way. In other words, even the authors of the only evidence you've brought to the table about "declining standards" won't say that standards have declined.

Allo V Psycho
Catbabel
Posts: 735
Joined: Sat Nov 16, 2019 8:18 am

Re: has education funding increased?

Post by Allo V Psycho » Thu Jan 16, 2020 5:13 pm

I think I’ll return to the de-rail topic of ‘have standards changed?’, just because I’m interested in it, and perhaps others are too.
As one might guess, it is quite complicated.

One could ask “Have questions (aka ‘items’) got easier?”. There is a way to explore this using Item Response Theory
https://en.wikipedia.org/wiki/Item_response_theory (which relies on a number of fairly plausible assumptions).

For each item, we can develop an Item Characteristic Curve (ICC), which shows the absolute difficulty of the item graphically. We could in principle compare the ICC for items administered in the past at different stages with those for current items (if we assume that candidate ability has remained unchanged: but see the Flynn effect https://en.wikipedia.org/wiki/Flynn_effect).

However, “Have items got easier (or more difficult)” is not the same question as “Have standards decreased (or increased)”. In fact, it is perfectly possible for standards to become more demanding, even when items become easier.

The first, and relatively trivial, way in which this may be the case, is if the level of performance required (e.g. the pass mark) is increased. In my day job, we routinely measure the difficulty of individual tests, and adjust the ‘pass score’ accordingly, since we are aiming for the same level of performance across a number of test administrations.

The second relates to validity (‘what the test is intended to measure’). An item may be difficult but also of low validity for the test intention. (‘True but trivial’ is one quite common example in testing). Validity itself is a complicated construct, but in general, I’m particularly interested in predictive validity: does the test predict later educational/work-place performance?

A test may therefore change to have items of lower abstract difficulty but disproportionately greater validity. In such a case, standards would have increased, even though item and test difficulty had decreased. This is similar to the situation Ken McKenzie described previously with regard to Physics degrees.

A test can also be designed to be sensitive in a particular area. This is shown by something called the Test Information Curve, also derived from Item Response Theory. We could design a test to be sensitive at the highest levels of performance, with many difficult items. But the top performing students are generally few in number. It is generally more useful to design a test to be sensitive around a critical boundary, such as the pass/fail boundary. In healthcare education, for instance, I’m more interested in the question “do those who pass, deserve to pass?” than “Have I ranked the top 5% correctly?”. I might therefore shift the mix of item difficulties to cluster around the pass/fail boundary, making the test as a whole technically 'easier' but more valid for my purpose.

Note: even the simple-seeming, but dubiously relevant, question of “has item difficulty decreased” for something like A levels, hasn’t been comprehensively answered. To calculate the ICC, a large number of candidate responses are needed. Four hundred is sometimes suggested as a minimum. The Jones et al paper we have discussed previously on this thread, managed to find just 66 candidate responses, at only 4 time points, and for only one subject. But I hope I’ve made it clear that item difficulty is not the key question.

I think I’ll pause there, and see if this has been of any interest to anyone. There is more I could say.

Oh boy, is there more I could say…

User avatar
bolo
Dorkwood
Posts: 1016
Joined: Mon Nov 11, 2019 1:17 pm
Location: Washington DC

Re: has education funding increased?

Post by bolo » Thu Jan 16, 2020 8:55 pm

Yes, this was an interesting read. Thank you for taking the time to share your expertise. I personally found this bit particularly significant:
Allo V Psycho wrote:
Thu Jan 16, 2020 5:13 pm
It is generally more useful to design a test to be sensitive around a critical boundary, such as the pass/fail boundary. In healthcare education, for instance, I’m more interested in the question “do those who pass, deserve to pass?” than “Have I ranked the top 5% correctly?”. I might therefore shift the mix of item difficulties to cluster around the pass/fail boundary, making the test as a whole technically 'easier' but more valid for my purpose.
I can imagine that identifying the critical boundary would be rather difficult for A levels, given the differences between universities.

sheldrake
After Pie
Posts: 1819
Joined: Fri Dec 20, 2019 2:48 am

Re: has education funding increased?

Post by sheldrake » Thu Jan 16, 2020 9:27 pm

plodder wrote:
Tue Jan 07, 2020 6:17 pm
No, it's yours. Allo is quoting the authors as saying you can't draw any reliable results either way. In other words, even the authors of the only evidence you've brought to the table about "declining standards" won't say that standards have declined.
Yes they did, that was the conclusion of the paper.

Allo V Psycho
Catbabel
Posts: 735
Joined: Sat Nov 16, 2019 8:18 am

Re: has education funding increased?

Post by Allo V Psycho » Thu Jan 16, 2020 11:26 pm

bolo wrote:
Thu Jan 16, 2020 8:55 pm
Yes, this was an interesting read. Thank you for taking the time to share your expertise. I personally found this bit particularly significant:
Allo V Psycho wrote:
Thu Jan 16, 2020 5:13 pm
It is generally more useful to design a test to be sensitive around a critical boundary, such as the pass/fail boundary. In healthcare education, for instance, I’m more interested in the question “do those who pass, deserve to pass?” than “Have I ranked the top 5% correctly?”. I might therefore shift the mix of item difficulties to cluster around the pass/fail boundary, making the test as a whole technically 'easier' but more valid for my purpose.
I can imagine that identifying the critical boundary would be rather difficult for A levels, given the differences between universities.
The critical boundaries are set by A-level exam boards, not by the Universities, but in general, you are right, it is difficult, and, in the end, is a matter of social consensus. There are various methods of promoting that social consensus, but none of them make it truly objective.

The boundaries are also fuzzy. On any given test, there are some candidates who pass who 'deserve' to pass and some who fail who 'deserve' to fail. But there are also some who fail who deserve to pass, and some who pass who deserve to fail. I could go into this in more detail, if anyone was interested?

User avatar
bolo
Dorkwood
Posts: 1016
Joined: Mon Nov 11, 2019 1:17 pm
Location: Washington DC

Re: has education funding increased?

Post by bolo » Thu Jan 16, 2020 11:29 pm

Allo V Psycho wrote:
Thu Jan 16, 2020 11:26 pm
I could go into this in more detail, if anyone was interested?
I'm interested.

Allo V Psycho
Catbabel
Posts: 735
Joined: Sat Nov 16, 2019 8:18 am

Re: has education funding increased?

Post by Allo V Psycho » Thu Jan 16, 2020 11:57 pm

bolo wrote:
Thu Jan 16, 2020 11:29 pm
Allo V Psycho wrote:
Thu Jan 16, 2020 11:26 pm
I could go into this in more detail, if anyone was interested?
I'm interested.
Well, how it is addressed depends on the relative costs of false positives (who pass but deserve to fail), and false negatives (who fail but deserve to pass). The cost of a false negative generally falls mainly on the individual (and their family): they have invested time and money in study, but not achieved they outcome they 'deserve'. The cost of a false positive may fall on members of society, and society in general. In healthcare education, this may appear as a patient safety issue.

One way of addressing this is by determining the consensus view of the 'pass mark' and then adding, say, 1 Standard Error of Measurement* to it. This will reduce the number of false positives, but increase the number of false negatives. Conversely, subtracting a Standard Error of Measurement decreases false negatives but increases false positives. Both strategies have been used in professional settings.

Another issue concerns re-sits. Re-sits can help address the problem of false negatives - on another administration of the test, the candidate who deserves to pass may well pass. But this approach is generally asymmetric: the false negatives are allowed to proceed even if they just scraped through, and are rarely tested again on that particular assessment.

Equally, by the operation of chance, multiple re-sit opportunity may mean that a true negative might pass given enough attempts. So one interesting question is, how many re-sits should be permitted? And exams are expensive to design, administer and score - who bears the cost of that, and of re-sits? The medical Royal Colleges and GMC charge hundreds, indeed, thousands, of pounds for exams and re-sits.

And we have hardly started on the complexities. You'll notice I have dichotomised candidates into true positives and negatives. But this is unrealistic too. Curricula must be designed, teaching delivered, assessments developed and scored, and standards set: and at each step, variables independent of the candidate may occur, and must be taken into account.

Of course, it all matters hugely, to the individuals involved, and to society. It is all extremely interesting, at least to me.


*https://www.statisticshowto.datascience ... asurement/

Post Reply