# Comparing observed and predicted values across several measurements

As a neuropsychology graduate student with some experience in statistics (I'm usually the guy other psychologists come to with statistics problems after trying it themselves but before seeing a statistician), I am investigating whether a medical treatment has an effect on cognitive measures (aside from curing the medical problem). I have been looking for a statistically sound method to approach the following problem (preferably using R) which will also pass muster with (medical) reviewers. I will try to explain the problem, which appears to be somewhat unusual - perhaps because of an error in my thinking.

I have data from two populations: a large group of controls, and a smaller group of patients (n ~ 20). In both groups, I have scores on a normally distributed cognitive scale at four time points (T0-T3), which in patients corresponds to a pre-treatment score (T0) and three post-treatment scores (T1-T3). As part of an accepted approach in neuropsychology (SRB-analysis; e.g. Duff, 2012), I have used multiple regression on the control data to model scores at T1, T2 and T3 from demographics and baseline (T0) scores. This regression model is then used to predict scores at T1-T3 in patients, and change in individual patients is evaluated based on the difference between predicted and observed score at T1-T3 (as well as the prediction error/variability in controls). For a single patient that looks something like this:

                   T0      T1      T2      T3
Observed Score     100     105     106     107
Predicted Score    -       110     117     120


However, the analysis is intended for an audience that is used to group (rather than individual) analysis, and to (partially) oblige this audience I would like to compare the predictions in patients as a group with the observed scores. This group analysis presents with me with two problems:

1. Due to phenomena such as regression to the mean, practice effects, etc., the distribution of the predicted scores is different from the distribution of the observed scores. For example, regression to the mean decreases the spread in the predicted scores (as compared to observed scores).
2. Although data exploration (histograms, qq-plots, etc) suggest that difference scores (observed-predicted) are normally distributed, the group size (n ~20) is smaller than what is usually recommended as a minimum group size for approximate normality (n ~30).

I would like to answer two questions in the group analysis:

1. Are observed scores different from (/lower than) predicted scores?
2. At which time points (T1-T3) are observed scores lower than predicted scores?

As a first approximation, I have used one-sample t-tests on the difference scores (observed-predicted) with Bonferroni correction for multiple (3) comparisons. But that is clearly a non-optimal solution: it does not directly provide the omnibus test required to answer the first question and it assumes normality where that is at least somewhat doubtful.

Hence my question: what would be the optimal way to perform the group analysis?

• I'm concerned about some of your reasoning/premises. Why would group-size relate to how near to normality the population distribution of difference sores might be? Recommended by whom? What did they actually recommend, and what justification did they offer? Jul 21 '16 at 3:44
• I'm sorry for being slow to reply, but I only realized today that there were new comments to this question. Oct 27 '16 at 14:00
• Since this was indeed posted long ago, I will add that we have since then published these data using the t-test solution offered in my original question. Oct 27 '16 at 14:17
• Many sources state this "rule" but the rule is so vague as to be of no value whatever - there is nothing special about 30 in any sense. In some cases n=2 is fine, in others n=200000 is not nearly enough (and we've seen at least one real example on site of actual cases where huge numbers like that were not sufficient). In fact the "rule" quite literally commits the fallacy of circular reasoning / begging the question (how do we know whether it's "too weird" other than by looking to see if the result were the case? If we have no other criterion then this "rule of 30" is contentless) Oct 27 '16 at 21:58
• Perhaps, it's at least plausible. If only all distributions were as nicely behaved as the binomial for $\frac13\leq p\leq\frac23$. Jan 15 at 22:07