sheldrake wrote: ↑Fri Jan 17, 2020 10:17 am
I see that's the case for a population with randomly distributed ability in their simulations, but the study I cited used experts.
Well:
1) The best method to show that items were easier would be to calculate the ICCs.
2) But there isn't enough data to calculate ICCs.
3) So Jones et al used a Comparative Judgement method (which is pretty cool, by the way).
4) But, like, no one can find a sufficient number of experts to make the comparative judgements. There may not even BE enough experts to make the comparative judgements in an informed way.
5) So they used 20 maths graduate students (because they were handy, presumably!). These are not experts in the curriculum used in the 60s, therefore they may interpret items which were not in their own curricula as more difficult.
6) In conclusion, we cannot
definitively conclude that standards in maths have fallen from this paper.
7) If it helps to have input from someone other than me, see Coe, R., 2010. Understanding comparability of examination standards. Research Papers in Education, 25(3), pp.271-284.
An example of this can be seen in discussions about whether standards in mathematics examinations have changed over time. Suppose at one time a mathematics examination assesses a range of ‘traditional’ skills, such as algebraic accuracy and speed and the application of standard algorithms and techniques. Over time it changes to incorporate some ‘modern’ skills such as open‐ended investigation, extended problem‐solving and modelling of real‐life problems, with a consequent lessening of emphasis on the ‘traditional’ skills. Although these two types of skills will overlap, they may also be seen as distinct constructs within the broad construct of mathematical ability. From the point of view of someone who values the ‘traditional’ skills, the ‘standard’ of the mathematics examination will be judged by the level of ‘traditional’ skills implied by a particular grade. From this perspective, relative to the ‘traditional’ skills construct, the ‘standard’ of the examination will be seen to have fallen. However, from the viewpoint of an advocate of ‘modern’ skills, the level of mathematical (i.e. ‘modern’) skill required to achieve a particular grade in the past was low and has risen appreciably. The examination has therefore become more demanding and the ‘standard’ has risen.
8) I am not Robert Coe.