6
$\begingroup$

I am facing the following situation. I asked my class of 19 students to rate their interest about a certain topic on a 1-5 (discrete) scale. I then spent some time teaching them about this topic for a few hours after which I asked them to answer the same question. I now have two sets of 19 values and also know the pairing between them.

What would be the proper way of testing whether or not my teaching sparked their interest?

I was gonna do a dependent t-test for paired samples, but I'm concerned that the data does not follow a normal distribution. First of all it is discrete, but I was told that it is not an issue as long as it passes the SPSS normality tests. Unfortunately my distributions don't pass those tests.

Can I still do the t-test? What would be the analogous test for discrete (but still ordered) data?

$\endgroup$

2 Answers 2

7
$\begingroup$

Lazy version: permutation testing.

The null hypothesis is that their interests would be unchanged. Operationally, that means that for each student, you might as well flip a coin to pick their "before" rating and "after" rating. So do that, a large number of times. If the result is more extreme than the one you have often, then you might as well believe the null hypothesis.

So something like this:

import random

# First of pair is before, second is after.
observed = [
    (3, 4), (2, 2), (3, 1), (4, 5), (5, 5),
    (2, 3), (4, 4), (3, 4), (3, 3), (2, 4)
]

def improvement(observed):
    "How many students' interest scores improve?"
    return sum(a > b for b, a in observed)

print(f'Observed improvement: {improvement(observed)}')

def assume_null(observed):
    "For each pair, toss coin to pick before and after."
    for b, a in observed:
        yield (a, b) if random.choice([True, False]) else (b, a)


# Number of simulations from the null hypothesis.
B = 1000

# Generate many potential improvements under the null hypothesis.
null_distr = [improvement(assume_null(observed)) for _ in range(0, B)]

# Fraction of simulations from the null hypothesis that are at least as extreme as
# the one observed in real life.
more_extreme = sum(impr >= improvement(observed) for impr in null_distr)
p = more_extreme/B

print(f'The null hypothesis gets at least as extreme a result in {p*100:.2f} % of the simulations.')

If you want to compute how much the scores improved, you just chuck that computation into the improvement function:

def improvement(observed):
    "By how many points did each student's interest improve?"
    return sum(a - b for b, a in observed)

If you have unpaired data, the approach is the same, but it will be much harder to show significance (because of the less powerful experiment design).

Instead of tossing a coin to pick a number from each pair, you just pool together both data sets (since under the null hypothesis, they are practically the same data set, right?) and then draw two random new data sets from that pool.

import random

before = [3, 2, 3, 4, 5, 2, 4, 3, 3, 2]
after = [4, 2, 1, 5, 5, 3, 4, 4, 3, 4]

def improvement(before, after):
    "By how much did interest change in aggregate?"
    return sum(after) - sum(before)

print(f'Observed improvement: {improvement(before, after)}')

def assume_null(before, after):
    "Draw two new before and after sets from both."
    n_b = len(before)
    n_a = len(after)
    combined_data = before + after
    shuffled = random.sample(combined_data, n_b + n_a)
    # Take the first half of the shuffled data to be "before" 
    # and the second half to be "after".
    return (shuffled[1:n_b], shuffled[n_b:n_b+n_a])


# Number of simulations from the null hypothesis.
B = 1000

# Generate many potential improvements under the null hypothesis.
null_distr = [improvement(*assume_null(before, after)) for _ in range(0, B)]

# Fraction of simulations from the null hypothesis that are at least as extreme as
# the one observed in real life.
more_extreme = sum(impr >= improvement(before, after) for impr in null_distr)
p = more_extreme/B

print(f'The null hypothesis gets at least as extreme a result in {p*100:.2f} % of the simulations.')

```
$\endgroup$
4
  • $\begingroup$ Does this procedure change if the results of both tests are anonymised? (I.e. You no longer have the pairs ${(3,3), (2,4), (5,3) \dots}$ but instead two multisets of results {3,2,5 ...), {3,4,3, ...}. $\endgroup$
    – sloth
    Jul 8, 2022 at 9:27
  • $\begingroup$ @sloth Wait, does that not mean the data is unpaired? $\endgroup$
    – kqr
    Jul 8, 2022 at 9:42
  • $\begingroup$ yes, unpaired is a better way of saying it! At least in the code above it seems that the data has to be paired? $\endgroup$
    – sloth
    Jul 8, 2022 at 9:59
  • 1
    $\begingroup$ @sloth I updated the answer with a version for unpaired data. $\endgroup$
    – kqr
    Jul 8, 2022 at 10:00
5
$\begingroup$

The sign test or the Wilcoxon sign-rank test present alternatives to a permutation test.

The former has really low power, but a reasonably comprehensible null and alternative hypothesis:

$\text{H}_{0}\text{: }P(X_{\text{before}} > X_{\text{after}}) = 0.5$ , with
$\text{H}_{\text{A}}\text{: }P(X_{\text{before}} > X_{\text{after}}) \ne 0.5$

In plain words, the null hypothesis of the sign test is that a randomly selected student is equally likely to have a before measurement that is larger than their after measurement.

The sign-rank test has much more power, but the null and alternate are less intuitive:

$\text{H}_{0}\text{: The distribution of } X \text{ is symmetrical and centered on 0,}$ with
$\text{H}_{\text{A}}\text{: The distribution of } X \text{ is either asymmetrical, or not centered on 0, or both.}$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.