For reasons of economy, a college has moved from live lectures to videotaped.

They have exam results from the year/years before the change, and the year/years after.

They want to be reassured that the new method delivers exam results THE SAME AS, OR NOT WORSE THAN, the old.

We can assume they have taken proper steps to have all the exams comparable.

How is this to be framed technically, and what tests can be applied?

  • 1
    $\begingroup$ This is called equivalence testing. Look at this topic for more info: stats.stackexchange.com/questions/139385/… $\endgroup$
    – alesc
    Apr 22 '15 at 7:35
  • $\begingroup$ Many thanks, alesc, especially for the topic name. I see a difficulty that distributions of exam results don't have exactly-paired values, but perhaps that can be overcome by grouping. Now to study the area. $\endgroup$
    – Bob
    Apr 24 '15 at 4:52
  • $\begingroup$ Do the exams have different questions each year, are the questions different or do they partly change? I ask because one thing that could be done is test equating: en.wikipedia.org/wiki/Equating that would make the tests have a common scale. However this needs you to have at least some questions to be answered by the same students or at least students with equivalent level of ability/knowledge. If you provide more detailed description I can write more on this method. $\endgroup$
    – Tim
    Apr 24 '15 at 10:31
  • $\begingroup$ Hi Tim; thanks for the suggestion - I read the Wikipedia article and I remember studying common item equating. That's what we have here - exam questions are drawn from pools of similar. Each year a fresh batch of students takes an equivalent exam. $\endgroup$
    – Bob
    Apr 26 '15 at 6:24
  • $\begingroup$ Hi @Tim; thanks - I read the Wikipedia article and remember studying common item equating. That's what we have here - exam questions drawn from pools of similar. Each year a fresh batch of students takes an equivalent exam. But I have to assume equivalent exams set to equivalent groups of students, ie that standardization has been done already by the subject specialists. The data I have is a sequence of distributions (ie the exam results for each year: how many students scored n%, n=1-100). If test equating can be applied to this, I'd be happy to know. $\endgroup$
    – Bob
    Apr 26 '15 at 6:35

The appropriate tests are in the book by Stefan Wellek (2010) Testing Statistical Hypotheses of Equivalence and Noninferiority, Second Edition: "6.2 Mann-Whitney test for equivalence" (pp126-135) and "7.5 A non-parametric k-sample test for equivalence" (pp 231-234).

The software package accompanying the book is at https://www.crcpress.com/downloads/K10408/TSHEQ_2ndEd_Programs_etc.zip. The software consists of scripts for the freeware R Statistical Package (equivalent SAS macros are also included). The script for the Mann-Whitney test for equivalence is called mawi. According to Wellek (2010) p128, the input data required for mawi are the significance level, the two sample sizes and the name of a file containing the raw data.

The original published source of the Mann-Whitney test for equivalence may be more convenient: Wellek, S. (1996), A New Approach to Equivalence Assessment in Standard Comparative Bioavailability Trials by Means of the Mann-Whitney Statistic. Biom. J., 38: 695–710. doi: 10.1002/bimj.4710380608 {onlinelibrary.wiley.com/doi/10.1002/bimj.4710380608/abstract}.

This extract from the abstract gives the idea of Wellek's approach:

"Let the two distribution functions be given by F(x) = P[X ≤ x], G(y) = P[Y ≤ y] with (X, Y) denoting an independent pair of real-valued random variables. An intuitively appealing way of putting the notion of equivalence of F and G into nonparametric terms can be based on the distance of the functional P[X > Y] from the value it takes if F and G coincide. This leads to the problem of testing the null hypothesis Ho P[X > Y] ≤ 1/2 - ε1 or P[X > Y] ≥ 1/2 + ε2 versus H1 : 1/2 − ε1 < P[X > Y] < 1/2 + ε2, with sufficiently small ε1, ε2 in (0, 1/2). The testing procedure we derive for (H0, H1), and propose to term the Mann-Whitney test for equivalence, consists of carrying out in terms of the U-statistics estimator of P[X > Y] the uniformly most powerful level a test for an interval hypothesis about the mean of a Gaussian distribution with fixed variance. The test is shown to be asymptotically distribution-free with respect to the significance level. In addition, results of an extensive simulation study are presented which suggest that the new test controls the level even with sample sizes as small as 10. For normally distributed data, the loss in power as against the optimal parametric procedure is found to be almost as small as in comparisons between the Mann-Whitney and the t-statistic in the conventional one or two-sided setting, provided the power of the parametric test does not fall short of 80%."

Clearly there are many other approaches to the problem, but Wellek's solution seems particularly suitable to this case.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.