13
$\begingroup$

Let's say I have a dataset to which I have estimated a relationship using non-parametric regression, specifically Kernel (obviously in this hypothetical example it's probably overfit slightly). The dataset is from x = 0 to x = 4. How would I go about extrapolating, and finding the y-value for say x = 4.2?

Would I simply extend the last connecting line of the regression, as depicted below? Or is this incorrect?

The reason I want to do this, is because I want to calculate LOOCV - which involves omitting each data point in turn, and I'm unclear on what to do when I omit the first last or data point in a range.

enter image description here

(Please be aware this is just randomly generated data, for illustrative purposes).

Thank you

Edit 1: Emphasis on reason for asking.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ You might want to consider highlighting the part where you explain the reason for asking, because neither answer seems to address it. Cross validation is used to assess model stability and get an idea of the out-of-sample error. If you only leave out observations away from the extremes, you will never get an idea of how well the model predicts observations near the extremes. $\endgroup$
    – Frans Rodenburg
    Commented Jun 8, 2021 at 6:14
  • $\begingroup$ Good suggestion, thank you, and for your secondary point also! $\endgroup$
    – TheFriendlyAsker
    Commented Jun 8, 2021 at 13:36

2 Answers 2

Reset to default
15
$\begingroup$

You can't. Machine learning models, broadly speaking, learn to interpolate between the data points. Here you are trying to extrapolate, extrapolation is hard and can easily go wrong. If you used a simple model, say linear regression, then it is easy to extrapolate and fairly easy to assess how bad the result could go. If you are using a more complicated model, say a polynomial regression, then as in the extract of the xkcd comic posted below, it can go arbitrarily bad.

Polynomial curve fitted to data going very wrong.

Models like random forest or kernel regression can only interpolate between the data points. To extrapolate beyound the data, you could do something like using the last seen slope (as you did on the plot attached to the question), or just use the prediction for the last known point (x=4) as predictions for everything above. Such simple methods for extrapolating are commonly used for time-series and may be preferred if you don't have much data. Both solutions are rather arbitrary and there is no good answer.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Thanks a lot for your response Tim, completely appreciate the clear write up. Is it therefore not recommended to calculate leave one out cross validation for a non-parametric function? For instance, if I were to exclude the first or last data point I would need to extrapolate when estimating the MSE at that value of x. Alternatively, do I just limit the cross validation to only the points which won’t require extrapolation? (I.e. Never exclude first or last point)? I’ve seen some people perform the cross validation in R, and just wondering what assumptions these functions make. $\endgroup$
    – TheFriendlyAsker
    Commented Jun 7, 2021 at 21:09
  • $\begingroup$ -1 Good answer, except that extrapolating in linear regression is also a no-no. $\endgroup$
    – Alexis
    Commented Jun 8, 2021 at 0:07
  • 5
    $\begingroup$ @Alexis I never said it’s a good idea to use linear regression to extrapolate. I said it’s easy and “fairly easy to assess how bad it is”. $\endgroup$
    – Tim
    Commented Jun 8, 2021 at 5:05
3
$\begingroup$

Typically I have seen that done just by extending the last line forward, although this obviously can be quite dangerous because of the nature of that local fitting and extreme values near the end points.

There may be some advanced techniques for it or you could take some notes from time-series literature and 'dampen' the last line or combine it with a global measure for the trend. So you use that last line averaged with the overall slope of a simple regression or something to try and keep your last local measure in line. These are all just off the cuff suggestions but no matter what you do extending it outside of your fitted x-value range is possible but dangerous.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.