Scrutable

Note: even the simple-seeming, but dubiously relevant, question of “has item difficulty decreased” for something like A levels, hasn’t been comprehensively answered. To calculate the ICC, a large number of candidate responses are needed. Four hundred is sometimes suggested as a minimum. The Jones et al paper we have discussed previously on this thread, managed to find just 66 candidate responses, at only 4 time points, and for only one subject. But I hope I’ve made it clear that item difficulty is not the key question.

Why is 400 suggested as a minimum, and by whom ?

Just one subject is sufficient, when discussing the difficulty of that subject.

sheldrake wrote: Fri Jan 17, 2020 12:00 am

Note: even the simple-seeming, but dubiously relevant, question of “has item difficulty decreased” for something like A levels, hasn’t been comprehensively answered. To calculate the ICC, a large number of candidate responses are needed. Four hundred is sometimes suggested as a minimum. The Jones et al paper we have discussed previously on this thread, managed to find just 66 candidate responses, at only 4 time points, and for only one subject. But I hope I’ve made it clear that item difficulty is not the key question.

Why is 400 suggested as a minimum, and by whom ?

Just one subject is sufficient, when discussing the difficulty of that subject.

Baur, T. and Lukes, D., 2009. An evaluation of the IRT models through Monte Carlo simulation. UW-L Journal of Undergraduate Research, 12(1-7).

In order to estimate the difficulty of an item, you need a sufficiently large sample to estimate the ability of the population.

(edit: a bit more detail)

I see that's the case for a population with randomly distributed ability in their simulations, but the study I cited used experts.

sheldrake wrote: Fri Jan 17, 2020 10:17 am I see that's the case for a population with randomly distributed ability in their simulations, but the study I cited used experts.

Well:

1) The best method to show that items were easier would be to calculate the ICCs.

2) But there isn't enough data to calculate ICCs.

3) So Jones et al used a Comparative Judgement method (which is pretty cool, by the way).

4) But, like, no one can find a sufficient number of experts to make the comparative judgements. There may not even BE enough experts to make the comparative judgements in an informed way.

5) So they used 20 maths graduate students (because they were handy, presumably!). These are not experts in the curriculum used in the 60s, therefore they may interpret items which were not in their own curricula as more difficult.

6) In conclusion, we cannot definitively conclude that standards in maths have fallen from this paper.

7) If it helps to have input from someone other than me, see Coe, R., 2010. Understanding comparability of examination standards. Research Papers in Education, 25(3), pp.271-284.

An example of this can be seen in discussions about whether standards in mathematics examinations have changed over time. Suppose at one time a mathematics examination assesses a range of ‘traditional’ skills, such as algebraic accuracy and speed and the application of standard algorithms and techniques. Over time it changes to incorporate some ‘modern’ skills such as open‐ended investigation, extended problem‐solving and modelling of real‐life problems, with a consequent lessening of emphasis on the ‘traditional’ skills. Although these two types of skills will overlap, they may also be seen as distinct constructs within the broad construct of mathematical ability. From the point of view of someone who values the ‘traditional’ skills, the ‘standard’ of the mathematics examination will be judged by the level of ‘traditional’ skills implied by a particular grade. From this perspective, relative to the ‘traditional’ skills construct, the ‘standard’ of the examination will be seen to have fallen. However, from the viewpoint of an advocate of ‘modern’ skills, the level of mathematical (i.e. ‘modern’) skill required to achieve a particular grade in the past was low and has risen appreciably. The examination has therefore become more demanding and the ‘standard’ has risen.

8) I am not Robert Coe.

I for one bemoan the standards in modern shoemaking.

plodder wrote: Fri Jan 17, 2020 3:34 pm I for one bemoan the standards in modern shoemaking.

Your last words on this?

Allo V Psycho wrote: Fri Jan 17, 2020 2:34 pm
5) So they used 20 maths graduate students (because they were handy, presumably!). These are not experts in the curriculum used in the 60s, therefore they may interpret items which were not in their own curricula as more difficult.

And I think that could be a fair interpretation, no?

The open question is; does this work equally both ways, or are modern syllabi basically a subset of the old material. The change in university 1st years and comments from admissions tutors suggests the latter to me, but I accept that you might want something more rigorous.

Must of bin easy

https://www.theguardian.com/education/2 ... tire-class

Indeed.

I'd be investigating whether he leaked the paper tbh.

sheldrake wrote: Thu Jan 23, 2020 3:43 am Indeed.

I'd be investigating whether he leaked the paper tbh.

Its not very nice to assume cheating. Also I doubt that he could get away with it. He’s built up a reputation over years and it would be very difficult to keep mass cheating quiet over a long period of time.

You might want to see whether the pupils are put into sets and he gets to teach the most able.

Gfamily wrote: Fri Jan 17, 2020 4:05 pm

plodder wrote: Fri Jan 17, 2020 3:34 pm I for one bemoan the standards in modern shoemaking.

Your last words on this?

I expect he thinks it's a load of old cobblers.

Well, well, well.

The government has finally admitted that average per pupil school funding has fallen in real terms since 2010 – despite years of claims that spending is at record levels.

New figures published today by the Department for Education (DfE) shows that per pupil funding remained "broadly flat" between 2010-11 and 2015-16.

It then fell by 4.2 per cent over 2016-17 and 2017-18, and has subsequently increased by 1.9 per cent to the current level.

https://www.tes.com/news/school-funding ... lly-admits

Shocked, I say, shocked that the Tories are lying

CF

As good a place as any, £9,250 a year for YouTube access:

https://www.theguardian.com/education/2 ... n-lectures

plodder wrote: Sun Feb 16, 2020 9:07 am As good a place as any, £9,250 a year for YouTube access:

https://www.theguardian.com/education/2 ... n-lectures

Don't be silly. Lectures aren't the only (or even main) thing that universities supply as part of a course.

Don’t you be silly. The infrastructure and staff time associated with face to face lectures is obviously one of the significant costs and this is obviously a cost-related exercise.

From talking to staff at other unis this is generally acute responses to "oh crap we over-recruited" rather than cynical "pile 'em high" exercise from the top. Last year apparently there was a f.ck up with one uni's clearing software. They were offering chemistry places to people who'd initially signed up for medicine but missed the grades (people can then swap if their grades are good enough), but their software wasn't telling them that chemistry was full but still kept allowing them to add people. Once you've offered and they've accepted you're contractually obliged to take them. They had waaaaaay more people than they should have. It'd be interesting to see what the upshot was: as they couldn't fit all the year in their labs at one time...

...and this was realised when? the day the students all turned up, or months before? "Contractually obliged" is a euphemism here - you can always buy your way out of a contract if needed. This was a deliberate strategy to accommodate the mistake.

Of course, we rest assured that not a penny of tuition fees is wasted. Not a penny.

plodder wrote: Mon Feb 17, 2020 10:31 am ...and this was realised when? the day the students all turned up, or months before? "Contractually obliged" is a euphemism here - you can always buy your way out of a contract if needed. This was a deliberate strategy to accommodate the mistake.

Of course, we rest assured that not a penny of tuition fees is wasted. Not a penny.

No, unis are bound by CMA and this really does matter. There is a very big fear of being on the wrong end of a CMA judgement. They are petrified of it.

As this happened in clearing - it'll be in the period between A-Level results and arrival, so about 6 weeks. The cost of buying off those (in this case about 100 students) plus the reputational damage - thousands of future students not applying as they don't trust you to honour your offer, versus let's see what we can do to mitigate this and have a few hundred students complain and maybe three years down the line you'll have a crap NSS return for that one department.

Universities aren't perfect. But neither are they the gung-ho profiteering parody you are looking to paint them as.

On the other hand, I do despise the "unconditional if you put us as firm (maybe with some backhanders)" that has been going on, but that mainly is in response to the current demographic dip - but that has also caused some of these problems as unis attempt to ensure they can cover costs by getting enough students in, but then over-offer. The average offer-to-conversion rate is anywhere from 10-20%. Too many conditional offers make their grade, well now you've got to find ways to accommodate them.

Am I completely missing what the problem is here?

Plant, (lecture theatres and labs, for instance) is fixed in the short and medium term. Long term investment to build new plant is, well, long term, and has to reflect confidence in long term shifts in student intentions.

Staff numbers are slightly more responsive, but are only responsive in the medium term (for full time academics). Advertising, interviewing and recruiting new staff takes time. What takes even more time is shifting staff resources to create the budget for new recruitment. If Medieval Scottish Poetry* courses become less popular, and you are sure this is a long term condition, you still have to make staff redundant, or introduce voluntary severance, to free the financial resources to fund the newly popular courses.

Student numbers are highly variable, over a very short term. Even without errors such as the ones flammable flower described, there can be unexpected swings at offer time and even more at clearing, that leave an excess of students over staff/facilities.

Under these circumstances, video-broadcast of lectures is an entirely sensible, indeed routine, resolution. The extra income needs to be invested in the less obvious things like extra advisers of studies and tutorial leaders (both of which can be responsive in short time periods). When I was a student at Glasgow in the grant funded 1970s, Chemistry lectures were operated on an alternating basis - you had an orange or green ticket that admitted you to the live or video lecture theatre on alternate days. The video lecture theatre had a technician to carry out any practical demonstrations required.

I've also worked in a number of HE environments where live/video lectures were an intended feature, not a bug. For instance, where the programme was spread around various locations, the live lecture would alternate with the broadcast version. It was common then to install hardware to make interactivity with the live lecturer possible - nowadays apps like Poll Everywhere or Mentimeter make this trivially easy, and are much better.

I thought the orignal Guardian article mentioned above was of poor quality, in apparently not being aware of these issues, and relying on a few vox pops, a 'reportedly', and some apparent general prejudices about student funding. Fire safety regulations have always prevented 'students being crammed in' to lecture theatres anywhere I have worked.

Yes, the general financial pressure on Universities does create real problems: I just don't think this particular Guardian article comes very close to evidencing what those problems are. The increase in unconditional offers unmatched to available places is one I've mentioned before. But there is a bind here, where Government forces Universities to behave like commercial institutions, then they are castigated for behaving like commercial institutions.

*we scientists used to tease a friend doing Medieval Scottish poetry ("who'll employ you!"). He landed a highly paid and interesting job with Reuters while we struggled on PhD stipends.

FlammableFlower wrote: Mon Feb 17, 2020 2:08 pm

plodder wrote: Mon Feb 17, 2020 10:31 am ...and this was realised when? the day the students all turned up, or months before? "Contractually obliged" is a euphemism here - you can always buy your way out of a contract if needed. This was a deliberate strategy to accommodate the mistake.

Of course, we rest assured that not a penny of tuition fees is wasted. Not a penny.

No, unis are bound by CMA and this really does matter. There is a very big fear of being on the wrong end of a CMA judgement. They are petrified of it.

As this happened in clearing - it'll be in the period between A-Level results and arrival, so about 6 weeks. The cost of buying off those (in this case about 100 students) plus the reputational damage - thousands of future students not applying as they don't trust you to honour your offer, versus let's see what we can do to mitigate this and have a few hundred students complain and maybe three years down the line you'll have a crap NSS return for that one department.

Universities aren't perfect. But neither are they the gung-ho profiteering parody you are looking to paint them as.

On the other hand, I do despise the "unconditional if you put us as firm (maybe with some backhanders)" that has been going on, but that mainly is in response to the current demographic dip - but that has also caused some of these problems as unis attempt to ensure they can cover costs by getting enough students in, but then over-offer. The average offer-to-conversion rate is anywhere from 10-20%. Too many conditional offers make their grade, well now you've got to find ways to accommodate them.

We were obviously writing at the same time!

FlammableFlower wrote: Mon Feb 17, 2020 2:08 pm But neither are they the gung-ho profiteering parody you are looking to paint them as.

No I'm not. My impression is they've got a lot in common with the public sector though, with guaranteed incomes, back-covering and lots of old-school incumbent managers who will work like crazy to protect the status quo.

Allo V Psycho wrote: Mon Feb 17, 2020 2:30 pm But there is a bind here, where Government forces Universities to behave like commercial institutions, then they are castigated for behaving like commercial institutions.

Which commercial institutions would cut quality whilst trying to keep hold of the customer's money? The only way that could work is if a) the contract is biased horribly against the customer, or b) they were a monopoly.

plodder wrote: Mon Feb 17, 2020 3:18 pm

Allo V Psycho wrote: Mon Feb 17, 2020 2:30 pm But there is a bind here, where Government forces Universities to behave like commercial institutions, then they are castigated for behaving like commercial institutions.

Which commercial institutions would cut quality whilst trying to keep hold of the customer's money? The only way that could work is if a) the contract is biased horribly against the customer, or b) they were a monopoly.

Pretty much all consumer facing companies do this.

f.cking hell, what a great example.