10
$\begingroup$

It frequently comes up in social science that variables that should be distributed in some way, say normally, end up having a discontinuity in their distribution around certain points.

For instance, if there are specific cutoffs such as "passing/failing" and if these measures are subject to distortion, there may be a discontinuity at that point.

One prominent example (cited below) comes student standardized test scores are normally distributed basically everywhere except at 60% where there is very little mass from 50-60% and an excessive of mass around 60-65%. This occurs in cases where teachers grade their own students exams. The authors investigate whether teachers are really helping students pass exams.

The most convincing evidence without a doubt comes from showing the graphs of a bell curve with a big discontinuity around different cutoffs for different tests. However, how would you go about developing a statistical test? They tried interpolation and then comparing fraction above or below and also a t-test on the fraction 5 points above and below the cutoff. While sensible, these are ad-hoc. Can anyone think of anything better?

Link: Rules and Discretion in the Evaluation of Students and Schools: The Case of the New York Regents Examinations http://www.econ.berkeley.edu/~jmccrary/nys_regents_djmr_feb_23_2011.pdf

Distribution of test scores, manipulable ones in black, note the sharp drop in density below the cutoff and corresponding rise above

$\endgroup$
1
  • $\begingroup$ Just to clarify - are you testing for a generic lack of, e.g., Normality, or for the presence of a discontinuity at a pre-specified point? Your example is of the latter, but of course any goodness-of-fit test, e.g., Anderson-Darling or Shapiro-Wilk for Normality will serve, although with a highly specific alternative you could construct more powerful tests. Also, in your graph above you evidently have a sample of thousands; would this be typical too? $\endgroup$
    – jbowman
    Dec 9 '11 at 23:00
6
+50
$\begingroup$

It is important to frame the question properly and to adopt a useful conceptual model of the scores.

The question

The potential cheating thresholds, such as 55, 65, and 85, are known a priori independently of the data: they do not have to be determined from the data. (Therefore this is neither an outlier detection problem nor a distribution fitting problem.) The test should assess the evidence that some (not all) scores just less than these thresholds were moved to those thresholds (or, perhaps, just over those thresholds).

Conceptual model

For the conceptual model, it is crucial to understand that the scores are unlikely to have a normal distribution (nor any other easily parameterized distribution). That is abundantly clear in the posted example and in every other example from the original report. These scores represent a mixture of schools; even if distributions within any school were normal (they are not), the mixture is not likely to be normal.

A simple approach accepts that there is a true score distribution: the one that would be reported except for this particular form of cheating. It is therefore a non-parametric setting. That seems too broad, but there are some characteristics of the score distribution that can be anticipated or observed in the actual data:

  1. The counts of scores $i-1$, $i$, and $i+1$ will be closely correlated, $1 \le i \le 99$.

  2. There will be variations in these counts around some idealized smooth version of the score distribution. These variations will typically be of a size equal to the square root of the count.

  3. Cheating relative to a threshold $t$ will not affect the counts for any score $i\ge t$. Its effect is proportional to the count of each score (the number of students "at risk" for being affected by cheating). For scores $i$ below this threshold, the count $c(i)$ will be reduced by some fraction $\delta(t-i)c(i)$ and this amount will be added to $t(i)$.

  4. The amount of change decreases with the distance between a score and the threshold: $\delta(i)$ is a decreasing function of $i=1,2,\ldots$.

Given a threshold $t$, the null hypothesis (no cheating) is that $\delta(1)=0$, implying $\delta$ is identically $0$. The alternative is that $\delta(1)\gt 0$.

Constructing a test

What test statistic to use? According to these assumptions, (a) the effect is additive in the counts and (b) the greatest effect will occur right around the threshold. This indicates looking at first differences of the counts, $c'(i) = c(i+1)-c(i)$. Further consideration suggests going one step further: under the alternative hypothesis, we expect to see a sequence of gradually depressed counts as the score $i$ approaches the threshold $t$ from below, then (i) a large positive change at $t$ followed by (ii) a large negative change at $t+1$. To maximize the power of the test, then, let's look at second differences,

$$c''(i) = c'(i+1) - c'(i) = c(i+2) - 2c(i+1) + c(i),$$

because at $i = t-1$ this will combine a largish negative decline $c(t+1)-c(t)$ with the negative of a large positive increase $c(t) - c(t-1)$, thereby magnifying the cheating effect.

I am going to hypothesize--and this can be checked--that the serial correlation of the counts near the threshold is fairly small. (Serial correlation elsewhere is irrelevant.) This implies that the variance of $c''(t-1) = c(t+1) - 2c(t) + c(t-1)$ is approximately

$$\text{var}(c''(t-1)) \approx \text{var}(c(t+1)) + (-2)^2\text{var}(c(t)) + \text{var}(c(t-1)).$$

I previously suggested that $\text{var}(c(i)) \approx c(i)$ for all $i$ (something that also can be checked). Whence

$$z = c''(t-1) / \sqrt{c(t+1) + 4c(t) + c(t-1)}$$

should approximately have unit variance. For large score populations (the posted one looks to be around 20,000) we can expect an approximately Normal distribution of $c''(t-1)$, too. Since we expect a highly negative value to indicate a cheating pattern, we easily obtain a test of size $\alpha$: writing $\Phi$ for the cdf of the standard Normal distribution, reject the hypothesis of no cheating at threshold $t$ when $\Phi(z) \lt \alpha$.

Example

For example, consider this set of true test scores, drawn iid from a mixture of three Normal distributions:

Histogram of true scores

To this I applied a cheating schedule at the threshold $t=65$ defined by $\delta(i) = \exp(-2 i)$. This focuses almost all cheating on the one or two scores immediately below 65:

Histogram of scores after cheating

To get a sense of what the test does, I computed $z$ for every score, not just $t$, and plotted it against the score:

Plot of Z

(Actually, to avoid troubles with small counts, I first added 1 to every count from 0 through 100 in order to calculate the denominator of $z$.)

The fluctuation near 65 is apparent, as is the tendency for all other fluctuations to be about 1 in size, consistent with the assumptions of this test. The test statistic is $z = -4.19$ with a corresponding p-value of $\Phi(z) = 0.0000136$, an extremely significant result. Visual comparison with the figure in the question itself suggests this test would return a p-value at least as small.

(Please note, though, that the test itself does not use this plot, which is shown to illustrate the ideas. The test looks only at the plotted value at the threshold, nowhere else. It would nevertheless be good practice to make such a plot to confirm that the test statistic really does single out the expected thresholds as loci of cheating and that all other scores are not subject to such changes. Here, we see that at all other scores there is fluctuation between about -2 and 2, but rarely greater. Note, too, that one need not actually compute the standard deviation of the values in this plot in order to compute $z$, thereby avoiding problems associated with cheating effects inflating the fluctuations at multiple locations.)

When applying this test to multiple thresholds, a Bonferroni adjustment of the test size would be wise. Additional adjustment when applied to multiple tests at the same time would also be a good idea.

Evaluation

This procedure cannot seriously be proposed for use until it is tested on actual data. A good way would be to take scores for one test and use a non-critical score for the test as threshold. Presumably such a threshold has not been subject to this form of cheating. Simulate cheating according to this conceptual model and study the simulated distribution of $z$. This will indicate (a) whether the p-values are accurate and (b) the power of the test to indicate the simulated form of cheating. Indeed, one could employ such a simulation study on the very data one is evaluating, providing an extremely effective way of testing whether the test is appropriate and what its actual power is. Because the test statistic $z$ is so simple, simulations will be practicable to do and fast to execute.

$\endgroup$
1
  • $\begingroup$ This test needs to be adjusted slightly because the expectation of $z$ is (approximately) proportional to the second derivative of the distribution. In the example, where the threshold is near a mode, that second derivative is near zero so there's no problem, but for a threshold in an area of high curvature (around 70 or 90 in the simulated data) the adjustment could be material. If I get a chance I'll edit this answer accordingly. $\endgroup$
    – whuber
    Dec 15 '11 at 16:26
1
$\begingroup$

I suggest fitting a model which explicitly predicts the dips and then showing that it significantly better fits the data than a naive one.

You need two components:

  • initial distribution of scores,
  • procedure of rechecking (honest or not) of scores when one fits below a threshold.

One possible model for a single threshold (of value $t$) is the following: $$p_{final}(s) = p_{initial}(s) - p_{initial}(s)m(s\rightarrow t)+ \delta(s=t)\sum_{s'=0}^{t-1}p_{initial}(s')m(s'\rightarrow t),$$ where

  • $p_{final}(s)$ - the probability distribution of the final score,
  • $p_{initial}(s)$ - the probability distribution if there were not thresholds,
  • $m(s'\rightarrow t)$ - the probability of manipulation score $s'$ into passing score $t$,
  • $\delta(s=t)$ is the Kronecker delta, i.e. 1 if $s=t$ and 0 otherwise.

Typically you cannot rise scores much. I would suspect exponential decay $m(s'\rightarrow t)\approx a q^{t-s'}$, where $a$ is the proportion of rechecked (manipulated) scores.

As the initial distribution you can try to use Poisson or Gaussian distribution. Of course it would be ideally to have the same test but for one group of teachers provide thresholds and for the other - no thresholds.

If there are more thresholds then one can apply the same formula but with corrections for each $t_i$. Perhaps $a_i$ would be different as well (e.g. as the difference between fail-pass can be more important that between two passing grades).

Notes:

  • Sometimes there are procedures of rechecking tests if there are just below passing grade. Then it is more difficult to say which instances were honest and which - not.
  • $m(s\rightarrow t)$ will surely depend on the type of test. For example if there are open questions, then some answers may be ambiguous and number of them depends on $s$ (so for low-scoring it may be easier to rise the score). Whereas for closed-choice test there should be little to no difference on the number of the correct and incorrect answers.
  • Sometimes the 'corrected' scores may be above $t$ - the instead of idealized $\delta(s=t)$ one may plug sth different.
$\endgroup$
2
  • $\begingroup$ I'm not sure that answers my exact question. In this case, we don't have the ability to re-check any exams. All that is observed is a distribution of final scores. The distribution is mostly normal. Except, around a certain cutoff point where we suspect manipulation, there is a break in the normal curve. If the null is that the curve would be "smooth" at that point, how can we test it against an alternative hypothesis where it is "bumpy" $\endgroup$
    – d_a_c321
    Dec 2 '11 at 22:07
  • $\begingroup$ I think I understate the question. My point was to: fit Gaussian (2 parameters) and calculate $X^2$, then fit $p_{final}$ (2 parameters for Gaussian + (t+1) parameters for the thresholds) and calculate its $X^2$. Calculating smoothness (e.g. in the form of $\sum_{s=0}^{99}|p(s+1)-p(s)|^2$) may be interesting but then it is important to check underlying assumptions and etc (e.g. for tests with a lot of question of for 2 points there may be fairly high "initial" jaggedness). If one has access to raw data (i.e. all answers, not only total scores) then there is even more room for testing... $\endgroup$ Dec 2 '11 at 23:40
1
$\begingroup$

I would split this problem into two subproblems:

  • Estimate the parameters of a distribution to fit the data
  • Perform outlier detection using the fitted distribution

There are various ways of tackling either of the subproblems.

It seems to me that a Poisson distribution would fit the data, if it were independently and identically distributed (iid), which of course we think it isn't. If we naively try to estimate the parameters of the distribution we will be skewed by the outliers. Two possible ways to overcome this are to use Robust Regression techniques, or a heuristic method such as cross-validation.

For the outlier detection there are again numerous approaches. The simplest is to use the confidence intervals from the distribution we fitted in stage 1. Other methods include bootstrap methods and Monte-Carlo approaches.

Although this won't tell you that there is a "jump" in the distribution, it will tell you whether there are more outliers than expected for the sample size.

A more complex approach would be to construct various models for the data, such as compound distributions, and use some kind of model comparison method (AIC/BIC) to determine which of the models is the best fit for the data. However if you are simply looking for "deviation from an expected distribution" then this seems like overkill.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.