Suppose task performance $y$ increases with trial number $x$ ($x = 1, 2, …, 10$), so that there is a practice effect. Let’s suppose the practice effect is linear. Subjects have a long break, and then provide another observation of $y$ (for $x = 11$).

I have reason to believe that subjects get better at the task during this break, beyond what would be expected by the linear practice effect. Therefore, I want to test whether average performance on the 11th trial is predicted by a linear model fit to the first 10 trials. What would be the correct procedure to test this hypothesis? I imagine it would be something like

  1. Fit a regression model of y on x for x = 1,2,…,10.
  2. Compute the conditional mean given $x = 11$ based on the model and call this $Y_m$.
  3. Compute the sample mean of $y$ for $x = 11$ and call this $Y_s$.
  4. Test for a difference in the above two quantities.

But how do I compute the standard error of the difference? Or should I just compute confidence intervals for both $Y_m$ and $Y_s$ and see if they overlap?


Update: Can I fit a linear regression to all trials $x = 1,2,...,11$, include a dummy variable for the 11th trial, and test the significance of the dummy variable term?

  • $\begingroup$ Call $\hat{\beta}, \sigma^{2}$ your estimated slope and error variance from the regression. Then your estimators are $Y_{m} = 11 \hat{\beta}$ and $Y_{s}$ is, as you said, the sample mean of all $y$'s with $x = 11$. We know ${\rm var}(Y_{m} - Y_{s}) = {\rm var}(Y_{m}) + {\rm var}(Y_{s}) - 2{\rm cov}(Y_{m},Y_{s})$. The two variances are simple to calculate: ${\rm var}(Y_{m}) = 121 {\rm se}(\hat{\beta})^{2} + \sigma^{2}$, which should be available from the regression output. ${\rm var}(Y_{s})$ is just the variance of a sample mean. The ${\rm cov}(Y_{m},Y_{s})$ term is rather puzzling though ... $\endgroup$
    – Macro
    Jul 23 '11 at 2:27
  • $\begingroup$ I was unsure what the covariance between the predicted and observed means would be as well. $\endgroup$
    – lockedoff
    Jul 24 '11 at 23:57
  • $\begingroup$ an asterisk sort of point: overlapping confidence intervals don't mean lack of significance. This is a common error! Schenker, N. & Gentleman, J.F. On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals. Am Stat 55, 182-186 (2001). Cumming, G. & Finch, S. Inference by Eye: Confidence Intervals and How to Read Pictures of Data. Am Psychol 60, 170-180 (2005). $\endgroup$
    – dmk38
    Jul 25 '11 at 0:38

Your 'Update' is actually very clever, but I think the estimate of the slope would be somewhat confounded with the estimated effect of your dummy variable. Sometimes such things are hard to avoid, but I suggest a slightly altered approach that doesn't have this problem. You could fit one model:

$$ Y = \beta_{0} + X \beta_{1} + \epsilon $$

to the full data set, including the 11th time point. Now let $X_{k}$ be the $k$'th "trial" value you used for $X$. You could also fit the model

$$ Y = \alpha_{0} + X' \alpha_{1} + X_{11} \alpha_{2} + \varepsilon $$

where $X_{k}' = X_{k}$ if $k \leq 10$ and 0 otherwise, and $X_{11} = X_{k}$ if $k = 11$ and 0 otherwise. So, you have a general slope governing the first 10 trials, and the 11th trial is allowed to have whatever effect it wants. Then, compare the two using the likelihood ratio test with 1 degree of freedom. Note if $\alpha_{1}=\alpha_{2}$ then you arrive back at the smaller model, so the smaller model is a sub-model, which is why the LRT is appropriate here.

If significant, this would indicate that the second (and larger) model, which allows the 11th time point to have a different effect, fits the data better, indicating that using the same slope to fit the 11th predictor value as you did for the first 10 is not sufficient.

Note: The LRT is only a valid tool here if the errors are normally distributed (assuming this is OLS regression).

  • $\begingroup$ Thanks! Your answer makes the solution very clear. It's essentially a broken stick regression, where the second basis function can only taken on two values (0 or 11), I think. Also, it will work cleanly with modifications I wanted to make but was worried about how to handle outside of the realm of a single model (random effects for repeated observations, a power transformation of the response to normalize the residuals). $\endgroup$
    – lockedoff
    Jul 25 '11 at 4:34

Think of the distribution of values predicted by your regression as your theoretical distribution. Your null hypothesis is that the trial 11 values match this theoretical distribution--more specifically, that the trial 11 mean is consistent with the theoretical mean. Since the theoretical distribution has a known mean and s, you can use a Z-test to assess whether the trial 11 mean significantly differs.

  • $\begingroup$ Why can you ignore the uncertainty in your estimation of the regression function? $\endgroup$
    – Macro
    Jul 23 '11 at 2:17
  • $\begingroup$ Good question. I stand corrected. $\endgroup$
    – rolando2
    Jul 23 '11 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.