Say I have the following data,
S1 + S2 + S3 = 100%
If I have this distribution across multiple classes and multiple years, how can I do a comparitive assesment ? A naive score would be a weighted mean like S1 + 2S2 + 3S3 (so a bias towards strong performers). Anybody has pointers on better estimation ?
First, the question on weightings. Most commonly I see data like this analysed by collapsing two of the categories, so what is reported is often "% of students who are proficient and above". This is actually equivalent to creating a weighted mean of 0S1 + 1S2 + 1S3 and using that as the performance indicator for a particular class-year combination. Funnily enough this is widely accepted whereas your suggested weighted mean of 1S1 + 2S2 + 3S3, which is no more arbitrary and which definitely gives a more nuanced picture of success, will immediately raise hackles.
In the end there is a substantive question that only those with an interest in the broader research and policy environment can answer, which is how important is it to exceed average proficiency as opposed to just achieve it? If achieving average is all that counts, then 0,1,1 is a good set of weights. If excellence is the aim then 1,2,3 is a better set of weights - but there is no statistical reason to not use 1,2,4 or 2,3,7 or any other preference....
A second challenge comes from what is meant by "average". If you are developing a system for a nation-wide education system you need to remember that the scale you are proposing is only a relative one - it cannot be used, for example to measure absolute progress even in an individual unit (because if the national average is going up or down, there is no reason for an observed change in the proportion of students below or above average). This may sound obvious, but a glance at media around the world will show it is often forgotten.
