Calculate the confidence that the data point is NOT explained by the regression

Question

I have $n$ independent variables $x_i$ and dependent variables $y_i$ with uncertainties for both $x$ and $y$. I did a linear regression to get a model $\hat y = \beta x$.

Now I want to use this regression model to determine when the underlying system dynamics (a.k.a. $\beta$) changed. I calculate the residuals of new points to see their deviation from my model. I can plot the residuals and see that at one point the residuals increase a lot.

But what means a lot? I wanted to calculate a probability that this residual cannot be explained by the variance of the data. I am thinking this is the confidence that something in the system has changed.

My thought was that I can calculate standard deviation about the regression with $$ s_r = \sqrt{\frac{\sum (y_i - \hat y)}{n - 2}}, $$ and then use the probability for each point using the normal probability density function.

But this seems very naïve and assumes that my regression curve is the actual true relationship. I am not taking into account that if the new values are far away from my first ones, the model should naturally be bad. This led me towards the confidence and prediction bands, but I do not know how I can use this to get the probability that the new point does not fit.

Also, I am completely neglecting the known uncertainties of my input.

Answer 1

On the one hand, you can absolutely use prediction intervals (your software should be able to do this by itself) and check whether a data point falls outside some (say) 99% PI. Of course, this can happen by chance (in 1% of cases even in the best of all possible worlds), and any detection of this kind will need to accept some false positives. You can reduce this by looking at multiple consecutive points outside the PI. This is close to statistical process control, googling for that may be helpful.

Yes, to a degree this assumes you know the true data generating process. But usually, being "close enough" is good enough. And after all, you are looking for points where the DGP changes so much that your understanding of it degrades noticeably - one could argue that for this to hold, your original understanding does not need to have been perfect, just that it used to be a lot better than it is after the event.

On the other hand, you can also take your ordered stream of residuals (it sure sounds like you have a time series, so you might also want to look at forecasting), and run standard structural change detection algorithms on those. Again, this is a large field, because there are many different "changes in structure", from level shifts (which may be what you mainly have in mind) to changes in trend or variance. Again, google for the term. In R, the strucchange package comes to mind.