Comparing pre and post test raw scores with one subject

Question

I am currently in the process of trying to analyze data from research carried out. I am analyzing whether a music intervention showed improvements in rate, accuracy, fluency, and comprehension on a single subject. I have used the Gray Oral Reading Test, which yields raw scores for rate, accuracy, fluency and comprehension. I want to compare those raw scores from the pretest in those four categories to the raw scores from the posttest in those four categories to see if there was a significant difference. I am not sure since I used only one subject if data analysis could be performed. If not what is appropriate to use in this case to compare the two sets of raw scores from each category?

Answer 1

I think whuber's comment about case studies rarely benefiting from formal testing is spot on. Still, as a thought exercise, I describe below some potential methods for formal testing one could consider.

Approach 1: There is a method to construct a confidence interval based on a single observation, which is described here: What can we say about population mean from a sample size of 1?

The method could in principle be used for hypothesis testing: If a 95% confidence interval excludes a particular value, $\theta$, then this is equivalent to rejecting the corresponding null hypothesis (that the true value is $\theta$) at $\alpha = .05$.

How could this be used in your case? Let $x$ be the amount of change on one of those 4 outcomes. Now you can use the method to construct a confidence interval for the true amount of change with $x \pm 9.68 |x|$ that has (at least) 95% coverage. If 0 is not included in that interval, then you could in principle reject the null hypothesis that the intervention has no effect.

However, that method will never allow you to reject that null hypothesis, since $x \pm 9.68 |x|$ will always include 0.

Approach 2: Here is another approach to consider. If the intervention has no effect, there is a 50/50 chance whether a score will go up or down by chance alone (for simplicity, let me exclude the possibility of exactly no change). But now suppose the scores improve on all four outcomes. If all four outcomes were independent, then we can easily calculate the chances of this happening under the null hypothesis (that the true chances of an increase are 50%) based on the binomial distribution: Namely $.5^4 = 0.0625$. That would be the one-sided p-value based on a sign test. That is not below $\alpha = .05$, so even in this most extreme case you would not be able to reject the null hypothesis at this conventionally used level. Moreover, this test assumes that the four outcomes are independent, which most surely is not the case.

Approach 3: Apparently, for rate, accuracy, fluency, and comprehension, the Gray Oral Reading Test provides "scaled scores having a mean of 10 and a standard deviation of 3." So let $x_B$ and $x_A$ denote the before and after scores on one of those outcomes. So, if the intervention has no effect, we could use $$z = \frac{x_A - x_B}{3 \sqrt{2(1-\rho)}}$$ as a test statistic to test if the true change is zero. Here, $\rho$ is the true correlation between before- and after-intervention scores on that particular outcome.

The denominator is the SD of the changes score, which is easy to derive from the fact that $Var(x_A - x_B) = Var(x_A) + Var(x_B) - 2Cov(x_A,x_B)$ and the covariance is $Cov(x_A,x_B) = \rho SD(x_A) SD(x_B)$. Here, we know that $Var(x_A) = Var(x_B) = 9$. The rest is simple algebra.

But you don't know $\rho$. Most surely, it will be positive, but you don't know how large it would be if you were to apply your intervention to a large sample of subjects. You can consider some 'what-if' scenarios, so, for example, if $\rho = .5$, then $z = (x_A - x_B) / 3$. For a one-sided z-test to be significant at $\alpha = .05$, $z$ must be equal to or larger than $1.645$. That would only be the case if $x_A - x_B \ge 3 \times 1.645 = 4.935$.

Assuming that we exclude the possibility that $\rho$ could be negative, the most conservative test is obtained when we assume that $\rho=0$. If you obtain $z \ge 1.645$ even then, you could argue that the null hypothesis of no improvement due to the intervention can be rejected. But this would require that the improvement is larger than $3 \times \sqrt{2} \times 1.645 = 6.979$, which would require a dramatic improvement due to the intervention.

Conclusion: But I would say that the there really isn't much you can do in terms of testing. And I don't think with a single subject, that should be the goal. I would just present the raw data (maybe in terms of a before vs. after graph) and just discuss what you see.