6
$\begingroup$

Using their previous results and the results form an aptitude test, students are given a target score. This score is a decimal number between $0$ and $120$. After their final exams they are graded on the same scale (although they receive a letter corresponding to intervals).

I would like to devise a statistical test to compare their target score with their actual score.

My idea is to use the $\chi^2$-test of significance, where the target scores are the expected frequencies $E_i$, and their actual scores are the observed frequencies $O_i$. Say that I have $20$ students, then $$X^2 = \sum_{i=1}^{20} \frac{(O_i-E_i)^2}{E_i}$$ My null hypothesis, $H_0$, would be that there is no difference in teaching and learning in my college than at the average college. (The target scores come from a team of government statisticians performing statistical analysis on all past outcomes, for all institutions.)

Clearly, if $O_i=E_i$ for all $1 \le i \le 20$ then $X^2=0$, and we can have $0\%$ confidence in rejecting the null hypothesis $H_0$. Let's say that we wanted to be $95\%$ confident that we could reject the null hypothesis, then we would find $\chi_{20}^2(5\%) = 31.41$ and require that $X^2 \ge 31.41$.

I get the degrees of freedom to be the same as the number of students because there are no constraints. The total final scores of the candidates, i.e. $\sum O_i$ isn't fixed.

Does this model seem like it would work, or is it a miss-application of the test?

Can you recommend any other models or any refinements to the current one?

$\endgroup$
  • 1
    $\begingroup$ "we can have 0% confidence in rejecting the null hypothesis" ... this is not the meaning of either an attained significance level or a p-value. Please avoid this construction. $\endgroup$ – Glen_b May 11 '14 at 21:10
  • $\begingroup$ Apart from the target scores, what else is available? Do you know the confidence intervals of target scores? $\endgroup$ – Aksakal May 12 '14 at 13:06
  • $\begingroup$ @Glen_b How would you phrase it then? $\endgroup$ – Fly by Night May 12 '14 at 16:56
  • $\begingroup$ @Aksakal The target scores are pretty tight. They come from hundreds of thousands of students, over many years. The target grades come from regression analysis done by a team of government statisticians. Besides, it doesn't really matter. We get judge against those targets no matter how tight the intervals. $\endgroup$ – Fly by Night May 12 '14 at 16:58
  • 1
    $\begingroup$ @Aksakal I'm not sure that I follow you. The target grades give, as I have said, what an "average student" in an "average college" will do. They have used the data from every single previous exam. They have used exam results from when the students where 11 years old, 16 years old and then a seperate apptitude test when they are 17 years old. Of course it is impossible to determinate a single student's performance. But the measure of success relates to a group of 40 students against the targets. I don't see how this duscussion is helping me answer my question. $\endgroup$ – Fly by Night May 12 '14 at 18:21
5
+25
$\begingroup$

1) The problem is that the chi-square arises because it's a sum of squares of standardized deviations of (approximately) normally distributed variables.

The numerator you propose is fine - under the null hypothesis it will be small. The problem arises with the denominator. In the case of sets of Poisson (or multinomial) counts, a sum of squares of standardized deviations will be (or will simplify to) dividing by the expected values.

The $E_i$ in the denominator of the chi-square doesn't seem to apply to your situation. To make it a chi-square test in your problem, you'd need to specify the variance of $O_i-E_i$.

You seem to be doing this on a per-student basis, so you'd need to have a variance-per-student. You might assume they have equal variance (which I doubt can be true, since the variability in scores getting near the limits of 0 and 120 will be smaller than the variability in scores when they're near the middle.

2) I am also concerned that your choice of statistic might not correspond to a question of interest. What is the underlying question you're trying to answer? Or, more directly, what are the alternatives you're most interested in being able to identify?

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The idea is this: my students have target scores, and then they achieve actual scores. The target scores have come for looking at all candidates, from all schools over many years. The target score tells me what an "average students", with the given student's past results, in an "average college" would achieve. If the results match the targets then we're an average college. The government return data that gives a percentage confidence that the "teaching and learning" in the college is better or worse than the average. I wanted to do something similar. $\endgroup$ – Fly by Night May 12 '14 at 16:43
2
$\begingroup$

You can't use $\chi^2$ test here, because it is for counts (frequency) data. $E_i$ in this test is the frequency of observing value $i$. In your case it is a single score of a student, i.e. the outcome of exactly one observation. The motivation for $\chi^2$ test is that you know the probability $P_i$ of an outcome $i$, then you conduct N experiments and observe $O_i$ number of outcome $i$, where $\sum_iO_i=N$, so you compare it to expected frequencies $E_i=N\times P_i$.

UPDATE: If this was USA, then the measurement errors of the test scores would have been available from College Board, they have statistical tables available, such as these. They claim that the measurement error is ~30 points. So, you can use this sort of information to see whether an individual student's score is different from the target score.

  1. You could also test whether the entire group of students scored differently than the target. In this case the standard deviation of the mean score of N students is $\sigma_N=\sigma/\sqrt{N}$. So, you can get the t-statistics by $t=\frac{\bar{T}-\bar{S}}{\sigma_N}$, where the numerator is a difference between an averages of target scores and the test results. Based on the t-stat you can say whether your test scores are significantly different from the target or not.

  2. In your case, you don't have the measurement error $\sigma$. You can try to estimate it under reasonable assumptions. The mechanics are simple: $\hat\sigma^2=Var[T_i-S_i]$, where $T_i,S_i$ - target and test scores of individual students. Basically, get the variance of the deviations from the target scores. This will give you an estimate of the measurement errors, which you can plug into the $\hat\sigma_N$ equation to get the estimate of the measurement error of the average class score similar to the first case.

Now, how would you interpret this result? Let's say that you got the class average lower than the target. Does it mean that you are teaching worse than the other schools? It would depend on how the scores are computed. For instance, if it is possible that all colleges had lower scores than the target in entire UK, then it would be possible that your college faired as well as others. On the other hand, if they somehow rescale the test scores so they match the target somehow in average UK, then it's a different story.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ "Of little use" may be too pessimistic. The question asks how to compare paired data of the form (predicted score, actual score). A t-test, for instance, would at least be able to falsify a null hypothesis that students on the average (within the college) scored as predicted, regardless of what is known about prediction errors. I suspect what motivates your comments is the realization that one could get more insight into the results by knowing what the prediction errors are (which is certainly true). $\endgroup$ – whuber May 12 '14 at 22:02
  • $\begingroup$ @whuber, I think that OP wants to compare his class to other schools, that's why I think that he can't do it based on test scores only. They get the targets in the beginning of the period, then they teach, and test. So the target score corresponds to a different period. $\endgroup$ – Aksakal May 12 '14 at 22:14
  • $\begingroup$ My take is that the O.P. understands that concern, as expressed in his reference to "teaching and learning," and that he is not making such a blunder as you describe. That is, I take it on faith that the objective is to find an effective, defensible way to compare a group of test scores to a paired group of predicted scores which are in any event going to be used as the basis of comparison. As he stated in a comment, "we judge against those targets." $\endgroup$ – whuber May 12 '14 at 22:19
  • 1
    $\begingroup$ @whuber You're exactly right. I'm trying to compare my students' scores to their target scores. I don't aim to compare schools against schools or years against years. $\endgroup$ – Fly by Night May 13 '14 at 13:42
1
$\begingroup$

I wonder if a simple rank sum test for stochastic dominance (or, if the assumptions of same shape and distributions differing only with respect to central location, test for median difference) would work. You have paired observations, and two measures that are not strictly normal (i.e. possible scores do not range from $-\infty$ to $\infty$). Seems a straightforward application. Added advantage that it is implemented in all the major software packages.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Well, I'm not sure, but you could wonder if the target score can predict the actual score. I think that a positive correlation between target and actual scores is a reasonable assumption, so you could try $O_i=\alpha + \beta E_i + \varepsilon$.

A toy example in R:

> set.seed(123)
> e <- rnorm(20, 80, 20)
> range(e)
[1]  40.67 115.74
> o <- e - rnorm(20, 20, 10)
> range(o)
[1]  21.29 111.17
> fit <- lm(o ~ e)
> summary(fit)
[...]
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   -22.73       8.51   -2.67    0.016 *  
e               1.04       0.10   10.38    5e-09 ***

In this example, you get: $$O_i = -\underset{(8.51)}{22.73}+\underset{(0.10)}{1.04}\;E_i+\varepsilon$$ (standard errors under the estimates.)

This would mean that:

  • actual scores are lesser than target scores by 22.7 on average;
  • high actual scores are slightly more likely when the target score is high.

If a regression doesn't look absurd to you, and if you get a reasonable explication (i.e., a reasonable $R^2$), you could add some predictors, e.g. gender.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I have no idea what the code means. $\endgroup$ – Fly by Night May 12 '14 at 16:54
  • $\begingroup$ Sorry. I've "translated" the code into an equation. $\endgroup$ – Sergio May 12 '14 at 17:13
  • $\begingroup$ @Sergio, suppose the intercept is negative, how does it help answer OP's question? It could mean that this year the testing had bias compared to target scores which were based on the previous scores. Without knowing the background info, it's impossible to interpret the results to compare this school to other schools. Also, how do you know that the variance of high scores is the same as of low scores? This is important in comparing schools in affluent and poor areas, as the mean score would be different to start with. $\endgroup$ – Aksakal May 12 '14 at 20:13
  • $\begingroup$ @Aksakal, a negative intercept would mean the the actual (not testing) score has bias compared to target (predicted) scores. Furthermore, if "students are given a target score" "using their previous results and the [I understand: their] results form an aptitude test", why should I compare schools in affluent ad poor areas? Furthermore, 'score' is one variable; why should I imagine multiple variances? If it were a multilevel model, I should shrink my estimates, but why bother if I'm comparing a single-school regression to a pooled regression? $\endgroup$ – Sergio May 12 '14 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.