I have a question about statistical significance in relation to confidence intervals from linear regression. I'm obviously far from a stats expert, and I've been searching for the answer to this, probably simple, question for a while now without any luck.
I've made an example to clarify my question: I'm interested in looking at the treatment effect of doing a change (e.g. spraying with pesticide) on one area, and use an untreated area for control. Before the treatment a "calibration line" is established between the two areas as the correlation between some observed response (e.g. annual crop yield, white open circles below, with regression line and 95% confidence intervals drawn)
[Fig: X and Y axis show the same response (e.g. annual crop yield), but for the control area(X axis) and for the treatment area (Y axis). Each data point is then the annual crop yield for both the control and the treatment area, a total of 10 years of data]
After the treatment this response is measured again (red open triangles), and the "treatment effect" is defined as the difference between the observed response in the treatment and the predicted response (the regression line).
My question is if you can say that the treatment effect is statistically significant if the data point is outside of the 95% confidence interval of the calibration regression? And why/why not ? (so in the plot example 3 of the observed treatment effects are significantly different from the predicted response to a 95% confidence level (p=0.05)?)
Edit1, additional question: Would prediction intervals, instead of confidence intervals, be more suitable to describe whether there has been a change in the relationship between the two areas after treatment?
Edit2: Would it be right so say that the confidence intervals can be used to check if the mean of the treatment effects are significantly different (and, as @Glen_b suggests, use the regression line/confidence line for the treatment points instead of single points. But when talking about whether a single sample is significantly different (as in my comment below to Glen_b) it is better to use the prediction interval?