# Comparing observed and predicted values across several measurements

As a neuropsychology graduate student with some experience in statistics (I'm usually the guy other psychologists come to with statistics problems after trying it themselves but before seeing a statistician), I am investigating whether a medical treatment has an effect on cognitive measures (aside from curing the medical problem). I have been looking for a statistically sound method to approach the following problem (preferably using R) which will also pass muster with (medical) reviewers. I will try to explain the problem, which appears to be somewhat unusual - perhaps because of an error in my thinking.

I have data from two populations: a large group of controls, and a smaller group of patients (n ~ 20). In both groups, I have scores on a normally distributed cognitive scale at four time points (T0-T3), which in patients corresponds to a pre-treatment score (T0) and three post-treatment scores (T1-T3). As part of an accepted approach in neuropsychology (SRB-analysis; e.g. Duff, 2012), I have used multiple regression on the control data to model scores at T1, T2 and T3 from demographics and baseline (T0) scores. This regression model is then used to predict scores at T1-T3 in patients, and change in individual patients is evaluated based on the difference between predicted and observed score at T1-T3 (as well as the prediction error/variability in controls). For a single patient that looks something like this:

                   T0      T1      T2      T3
Observed Score     100     105     106     107
Predicted Score    -       110     117     120


However, the analysis is intended for an audience that is used to group (rather than individual) analysis, and to (partially) oblige this audience I would like to compare the predictions in patients as a group with the observed scores. This group analysis presents with me with two problems:

1. Due to phenomena such as regression to the mean, practice effects, etc., the distribution of the predicted scores is different from the distribution of the observed scores. For example, regression to the mean decreases the spread in the predicted scores (as compared to observed scores).
2. Although data exploration (histograms, qq-plots, etc) suggest that difference scores (observed-predicted) are normally distributed, the group size (n ~20) is smaller than what is usually recommended as a minimum group size for approximate normality (n ~30).

I would like to answer two questions in the group analysis:

1. Are observed scores different from (/lower than) predicted scores?
2. At which time points (T1-T3) are observed scores lower than predicted scores?

As a first approximation, I have used one-sample t-tests on the difference scores (observed-predicted) with Bonferroni correction for multiple (3) comparisons. But that is clearly a non-optimal solution: it does not directly provide the omnibus test required to answer the first question and it assumes normality where that is at least somewhat doubtful.

Hence my question: what would be the optimal way to perform the group analysis?

• I'm concerned about some of your reasoning/premises. Why would group-size relate to how near to normality the population distribution of difference sores might be? Recommended by whom? What did they actually recommend, and what justification did they offer? – Glen_b Jul 21 '16 at 3:44
• I'm sorry for being slow to reply, but I only realized today that there were new comments to this question. – Ishisht Oct 27 '16 at 14:00
• Yeah, sorry to come in on it so long after you posted. – Glen_b Oct 27 '16 at 14:06
• Lost my edit. Regarding normality, the "rule of 30" is quoted in many sources, e.g., here: math.uah.edu/stat/sample/CLT.html. The reasoning as I understand it is that with distributions that are not too weird, with n>30 the central limit theorem implies that a normal approximation can reasonably be used. Of course you are right that sample size has no bearing on whether scores actually follow a normal distribution in the population. – Ishisht Oct 27 '16 at 14:15
• Since this was indeed posted long ago, I will add that we have since then published these data using the t-test solution offered in my original question. – Ishisht Oct 27 '16 at 14:17

## 1 Answer

GLM or general linear model, an option in SPSS, offers repeated measures analysis. Repeated measures analysis is useful for stacked data.

Repeated measures analysis is usually done with stacked data. Under repeated meaures, each observation, rather than each subject, is used as a case by SPSS.

Since you require something like a pairwise comparison, rather than a mere comparison of means, this option may be best suited for you.