3
$\begingroup$

My question is a relatively basic one. Say I have some sort of a model which appears to fit my data well, how confident should I be in predictions that my model make for inputs that are very different from what was used to fit the data?

Take the very basic problem below. Here I fit a straight line through a bunch of points that are tightly grouped around 1 on the x-axis. I would expect this model to give reasonable results for values of the independent variable that lie reasonably close to the ones in the sample used to fit the data, say between 0.8 and 1.2, but would be much less confident for a model reading where x=20.

Example one

Similarly, for the example below, I would be more worried about model predictions at points on the x-axis where my sample is sparsely populated or unpopulated (say at x=1 or x=2), than I would be in cases where there were many observations around that reading for x (say around 0.45).

Example two

Although the examples I am using here are a bit crude and much simpler than the problem I am trying to solve, they illustrate the issue. I am trying to understand the pitfalls of trusting my model for a set of independent variables that are atypical relative to the sample that my model was fitted on. I am trying to understand whether there is a way of quantifying the uncertainty of model readings at these points. I would also like to know whether there are any rules of thumb of how to treat these sorts of problems. If you know of any academic articles that discuss this issue I would greatly appreciate it if you could point them out to me.

$\endgroup$
  • 3
    $\begingroup$ I think you are looking for the concept of a prediction interval, at least for your first case. This takes the form of a band bounded above and below by hyperbolas so has the property which I expect you would accept that the interval gets wider as you move further away from the centroid. $\endgroup$ – mdewey Aug 2 '17 at 12:40
1
$\begingroup$

I am trying to understand whether there is a way of quantifying the uncertainty of model readings at these points.

If you plot the confidence interval for predictions at each point in the X-axis, you will get a hyperbola as @mdewey points out in the comments. So the further from your data mass, the wider the confidence interval on your prediction.

I would also like to know whether there are any rules of thumb of how to treat these sorts of problems.

It depends on how you fitted your model. If you derived the model parameters from the data (in your example, you calculated the slope and the intercept via a linear regression), then yes, extrapolation can lead to bad predictions. That is not necessarily the case, e.g. your results agree with previous literature on the matter, but additional validation is often needed in order to check if your model works outside the data mass and to narrow the wide confidence interval far from it.

On the other hand, if the parameters are calculated separately (for example, there is some fundamental law of nature that dictates the slope and intercept), and the data is only used to check if they are correct, then you should be as confident in extrapolating outside the data mass as you are in the method used for calculating the parameters in the first place.

Please note that, depending on your field of work, there might be standard practices in place for dealing with extrapolation. As an example, in the field of manufacturing oilwell casing strings, ISO 13679 is a widely adopted standard practice which prescribes that extrapolation outside of the testing envelope is not allowed.

$\endgroup$
  • $\begingroup$ Thanks for your thoughts, there are some very useful ideas there. The field in question, by the way, is finance. And the parameters come from the data, which as you point out, absolutely make extrapolation a problem. I will do some work to try and come up with confidence intervals to see whether the model then becomes more workable. $\endgroup$ – Rene Aug 3 '17 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.