How do I test whether an extrapolated mean for a regression model differs from an observed mean?

Suppose task performance $y$ increases with trial number $x$ ($x = 1, 2, …, 10$), so that there is a practice effect. Let’s suppose the practice effect is linear. Subjects have a long break, and then provide another observation of $y$ (for $x = 11$).

I have reason to believe that subjects get better at the task during this break, beyond what would be expected by the linear practice effect. Therefore, I want to test whether average performance on the 11th trial is predicted by a linear model fit to the first 10 trials. What would be the correct procedure to test this hypothesis? I imagine it would be something like

1. Fit a regression model of y on x for x = 1,2,…,10.
2. Compute the conditional mean given $x = 11$ based on the model and call this $Y_m$.
3. Compute the sample mean of $y$ for $x = 11$ and call this $Y_s$.
4. Test for a difference in the above two quantities.

But how do I compute the standard error of the difference? Or should I just compute confidence intervals for both $Y_m$ and $Y_s$ and see if they overlap?

--

Update: Can I fit a linear regression to all trials $x = 1,2,...,11$, include a dummy variable for the 11th trial, and test the significance of the dummy variable term?

• Call $\hat{\beta}, \sigma^{2}$ your estimated slope and error variance from the regression. Then your estimators are $Y_{m} = 11 \hat{\beta}$ and $Y_{s}$ is, as you said, the sample mean of all $y$'s with $x = 11$. We know ${\rm var}(Y_{m} - Y_{s}) = {\rm var}(Y_{m}) + {\rm var}(Y_{s}) - 2{\rm cov}(Y_{m},Y_{s})$. The two variances are simple to calculate: ${\rm var}(Y_{m}) = 121 {\rm se}(\hat{\beta})^{2} + \sigma^{2}$, which should be available from the regression output. ${\rm var}(Y_{s})$ is just the variance of a sample mean. The ${\rm cov}(Y_{m},Y_{s})$ term is rather puzzling though ... Jul 23 '11 at 2:27
• I was unsure what the covariance between the predicted and observed means would be as well. Jul 24 '11 at 23:57
• an asterisk sort of point: overlapping confidence intervals don't mean lack of significance. This is a common error! Schenker, N. & Gentleman, J.F. On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals. Am Stat 55, 182-186 (2001). Cumming, G. & Finch, S. Inference by Eye: Confidence Intervals and How to Read Pictures of Data. Am Psychol 60, 170-180 (2005). Jul 25 '11 at 0:38

Your 'Update' is actually very clever, but I think the estimate of the slope would be somewhat confounded with the estimated effect of your dummy variable. Sometimes such things are hard to avoid, but I suggest a slightly altered approach that doesn't have this problem. You could fit one model:

$$Y = \beta_{0} + X \beta_{1} + \epsilon$$

to the full data set, including the 11th time point. Now let $X_{k}$ be the $k$'th "trial" value you used for $X$. You could also fit the model

$$Y = \alpha_{0} + X' \alpha_{1} + X_{11} \alpha_{2} + \varepsilon$$

where $X_{k}' = X_{k}$ if $k \leq 10$ and 0 otherwise, and $X_{11} = X_{k}$ if $k = 11$ and 0 otherwise. So, you have a general slope governing the first 10 trials, and the 11th trial is allowed to have whatever effect it wants. Then, compare the two using the likelihood ratio test with 1 degree of freedom. Note if $\alpha_{1}=\alpha_{2}$ then you arrive back at the smaller model, so the smaller model is a sub-model, which is why the LRT is appropriate here.

If significant, this would indicate that the second (and larger) model, which allows the 11th time point to have a different effect, fits the data better, indicating that using the same slope to fit the 11th predictor value as you did for the first 10 is not sufficient.

Note: The LRT is only a valid tool here if the errors are normally distributed (assuming this is OLS regression).

• Thanks! Your answer makes the solution very clear. It's essentially a broken stick regression, where the second basis function can only taken on two values (0 or 11), I think. Also, it will work cleanly with modifications I wanted to make but was worried about how to handle outside of the realm of a single model (random effects for repeated observations, a power transformation of the response to normalize the residuals). Jul 25 '11 at 4:34

Think of the distribution of values predicted by your regression as your theoretical distribution. Your null hypothesis is that the trial 11 values match this theoretical distribution--more specifically, that the trial 11 mean is consistent with the theoretical mean. Since the theoretical distribution has a known mean and s, you can use a Z-test to assess whether the trial 11 mean significantly differs.

• Why can you ignore the uncertainty in your estimation of the regression function? Jul 23 '11 at 2:17
• Good question. I stand corrected. Jul 23 '11 at 12:09