# Significant difference from regression confidence intervals

I have a question about statistical significance in relation to confidence intervals from linear regression. I'm obviously far from a stats expert, and I've been searching for the answer to this, probably simple, question for a while now without any luck.

I've made an example to clarify my question: I'm interested in looking at the treatment effect of doing a change (e.g. spraying with pesticide) on one area, and use an untreated area for control. Before the treatment a "calibration line" is established between the two areas as the correlation between some observed response (e.g. annual crop yield, white open circles below, with regression line and 95% confidence intervals drawn) [Fig: X and Y axis show the same response (e.g. annual crop yield), but for the control area(X axis) and for the treatment area (Y axis). Each data point is then the annual crop yield for both the control and the treatment area, a total of 10 years of data]

After the treatment this response is measured again (red open triangles), and the "treatment effect" is defined as the difference between the observed response in the treatment and the predicted response (the regression line).

My question is if you can say that the treatment effect is statistically significant if the data point is outside of the 95% confidence interval of the calibration regression? And why/why not ? (so in the plot example 3 of the observed treatment effects are significantly different from the predicted response to a 95% confidence level (p=0.05)?)

Thanks

Edit1, additional question: Would prediction intervals, instead of confidence intervals, be more suitable to describe whether there has been a change in the relationship between the two areas after treatment?

Edit2: Would it be right so say that the confidence intervals can be used to check if the mean of the treatment effects are significantly different (and, as @Glen_b suggests, use the regression line/confidence line for the treatment points instead of single points. But when talking about whether a single sample is significantly different (as in my comment below to Glen_b) it is better to use the prediction interval?

• I thought this was a very well stated question. Clear and concise. +1 for that! Apr 15, 2013 at 9:09

You don't compare the individual points to conclude a treatment effect. You see whether the lines for the treatment and control are different.

In some circumstances, the fitted lines might be parallel, and just the difference in intercept is of interest. In others, both the intercept and slope might differ, and any difference would be of interest.

Testing point vs line in ordinary regression (not errors-in-variables, which is more complicated):

It's not correct to check if data values for another are in the confidence interval because the data values themselves have noise.

Call the first sample $(\underline{x}_1,\underline{y}_1)$, and the second one $(\underline{x}_2,\underline{y}_2)$. Your model for the first sample is $y_1(i) = \alpha_1 + \beta_1 x_{1,i} + \varepsilon_i$, with the usual iid $N(0,\sigma^2)$ assumption on the errors.

You want to see if a particular point $(x_{2,j},y_{2,j})$ is consistent with the first sample. Equivalently, to check whether an interval for $y_{2,j} - \left(\alpha_1 + \beta_1 x_{2,j}\right)$ includes 0 (notice the points are second-sample, the line is first-sample).

The usual way to obtain such CI would to construct a pivotal quantity, though one could simulate or boostrap as well.

However, since in this illustration we're doing it for a single point, under normal assumptions and with ordinary regression conditions, we can save some effort: this is a solved problem. It corresponds to (assuming sample 1 and sample 2 have a common population variance) checking whether one of the sample 2 observations lies within a prediction interval based on sample 1, rather than a confidence interval.

• Thanks for the input, and I agree that the lines should also be compared! I think that comparing the individual points does make sense in some cases, such as when the treatment effect is expected to decrease between each sample (for example habitat restoration after disturbance). So asking the question: was there a significant treatment effect 1 year after treatment? 2 years after treatment etc...to see when the habitat has been restored. Apr 10, 2013 at 11:57
• "such as when the treatment effect is expected to decrease between each sample" -- I'd have thought the relevant thing to do would be to model that decreasing effect (with the reduction in noise/uncertainty) rather than base it off individual, noisier points. You could do it point by point, but you wouldn't do that the way you suggest because (if I understood it right), it ignores the noise in the treatment sample. Apr 10, 2013 at 12:59
• Thanks again, using the trend is for sure a better way to go about this. I'm still curious though: IF (despite being a poor option) you would compare the single post-treatment points to the regression line, can you say anything about the significance of the difference based on confidence intervals (or prediction intervals) ? Apr 10, 2013 at 13:53
• What you'd do to compare an individual treatment point to a control group line is either construct a CI for the difference (to see if it contained zero) or do a hypothesis test directly; the two will involve pretty much the same calculation organized differently. There's something I have failed to comprehend - and it is important now: Consider the third black point (counting from the left). What exactly do its x- (horizontal axis) and y- values represent? What about the third red triangle? What do its x- and y- values represent? Apr 10, 2013 at 14:23
• Good tips, thanks! Perhaps I didn't clarify well enough in the question: The X and Y axis are responses that can be measured from the two areas, control and treatment area. As an example, this can be annual sediment load from two catchments/watersheds (so one catchment is control, one is treatment). The black points are data collected before the treatment and are used for the calibration line. Then a change (e.g. burned the forest) is made to the treatment area and we observe a change in response: the triangles. Hope that helped Apr 10, 2013 at 14:46