As a neuropsychology graduate student with some experience in statistics (I'm usually the guy other psychologists come to with statistics problems after trying it themselves but before seeing a statistician), I am investigating whether a medical treatment has an effect on cognitive measures (aside from curing the medical problem). I have been looking for a statistically sound method to approach the following problem (preferably using R) which will also pass muster with (medical) reviewers. I will try to explain the problem, which appears to be somewhat unusual - perhaps because of an error in my thinking.
I have data from two populations: a large group of controls, and a smaller group of patients (n ~ 20). In both groups, I have scores on a normally distributed cognitive scale at four time points (T0-T3), which in patients corresponds to a pre-treatment score (T0) and three post-treatment scores (T1-T3). As part of an accepted approach in neuropsychology (SRB-analysis; e.g. Duff, 2012), I have used multiple regression on the control data to model scores at T1, T2 and T3 from demographics and baseline (T0) scores. This regression model is then used to predict scores at T1-T3 in patients, and change in individual patients is evaluated based on the difference between predicted and observed score at T1-T3 (as well as the prediction error/variability in controls). For a single patient that looks something like this:
T0 T1 T2 T3 Observed Score 100 105 106 107 Predicted Score - 110 117 120
However, the analysis is intended for an audience that is used to group (rather than individual) analysis, and to (partially) oblige this audience I would like to compare the predictions in patients as a group with the observed scores. This group analysis presents with me with two problems:
- Due to phenomena such as regression to the mean, practice effects, etc., the distribution of the predicted scores is different from the distribution of the observed scores. For example, regression to the mean decreases the spread in the predicted scores (as compared to observed scores).
- Although data exploration (histograms, qq-plots, etc) suggest that difference scores (observed-predicted) are normally distributed, the group size (n ~20) is smaller than what is usually recommended as a minimum group size for approximate normality (n ~30).
I would like to answer two questions in the group analysis:
- Are observed scores different from (/lower than) predicted scores?
- At which time points (T1-T3) are observed scores lower than predicted scores?
As a first approximation, I have used one-sample t-tests on the difference scores (observed-predicted) with Bonferroni correction for multiple (3) comparisons. But that is clearly a non-optimal solution: it does not directly provide the omnibus test required to answer the first question and it assumes normality where that is at least somewhat doubtful.
Hence my question: what would be the optimal way to perform the group analysis?