How do I test whether an extrapolated mean for a regression model differs from an observed mean?

Question

Suppose task performance $y$ increases with trial number $x$ ($x = 1, 2, …, 10$), so that there is a practice effect. Let’s suppose the practice effect is linear. Subjects have a long break, and then provide another observation of $y$ (for $x = 11$).

I have reason to believe that subjects get better at the task during this break, beyond what would be expected by the linear practice effect. Therefore, I want to test whether average performance on the 11th trial is predicted by a linear model fit to the first 10 trials. What would be the correct procedure to test this hypothesis? I imagine it would be something like

1. Fit a regression model of y on x for x = 1,2,…,10.
2. Compute the conditional mean given $x = 11$ based on the model and call this $Y_m$.
3. Compute the sample mean of $y$ for $x = 11$ and call this $Y_s$.
4. Test for a difference in the above two quantities.

But how do I compute the standard error of the difference? Or should I just compute confidence intervals for both $Y_m$ and $Y_s$ and see if they overlap?

--

Update: Can I fit a linear regression to all trials $x = 1,2,...,11$, include a dummy variable for the 11th trial, and test the significance of the dummy variable term?

Answer 1

Your 'Update' is actually very clever, but I think the estimate of the slope would be somewhat confounded with the estimated effect of your dummy variable. Sometimes such things are hard to avoid, but I suggest a slightly altered approach that doesn't have this problem. You could fit one model:

$$ Y = \beta_{0} + X \beta_{1} + \epsilon $$

to the full data set, including the 11th time point. Now let $X_{k}$ be the $k$'th "trial" value you used for $X$. You could also fit the model

$$ Y = \alpha_{0} + X' \alpha_{1} + X_{11} \alpha_{2} + \varepsilon $$

where $X_{k}' = X_{k}$ if $k \leq 10$ and 0 otherwise, and $X_{11} = X_{k}$ if $k = 11$ and 0 otherwise. So, you have a general slope governing the first 10 trials, and the 11th trial is allowed to have whatever effect it wants. Then, compare the two using the likelihood ratio test with 1 degree of freedom. Note if $\alpha_{1}=\alpha_{2}$ then you arrive back at the smaller model, so the smaller model is a sub-model, which is why the LRT is appropriate here.

If significant, this would indicate that the second (and larger) model, which allows the 11th time point to have a different effect, fits the data better, indicating that using the same slope to fit the 11th predictor value as you did for the first 10 is not sufficient.

Note: The LRT is only a valid tool here if the errors are normally distributed (assuming this is OLS regression).

Answer 2

Think of the distribution of values predicted by your regression as your theoretical distribution. Your null hypothesis is that the trial 11 values match this theoretical distribution--more specifically, that the trial 11 mean is consistent with the theoretical mean. Since the theoretical distribution has a known mean and s, you can use a Z-test to assess whether the trial 11 mean significantly differs.