3
$\begingroup$

Those lines are based on a lowess fit of points I have.

I would like to be able to extrapolate those lines up to x=500 for instance. I would like to extend this plot and see if those lines reach a plateau or not.

Is there a way to do this ?

The CSV data for the blue line is here: http://db.tt/GHhAGLtM

enter image description here

The idea is fill this plot with extrapolated dashed lines. enter image description here

$\endgroup$

migrated from stackoverflow.com Apr 25 '12 at 16:15

This question came from our site for professional and enthusiast programmers.

  • 8
    $\begingroup$ Extrapolation of a fitted curve is something one should only do very carefully. Your extrapolation is only as good as the model that you used to fit the curve in the first place. If you have a mechanistic model in mind (i.e. this is some natural world phenomenon and you can model it from sound theoretical principles) then you can do it. Otherwise here be dragons. If you want some more help, please describe in words what the data means. $\endgroup$ – Andrie Apr 25 '12 at 11:47
  • 6
    $\begingroup$ Voted off-topic as this is a statistics question and can hence be better asked at crossvalidated.com. Short answer: You can't extrapolate from a lowess curve, as that is a LOCAL regression. You need to use different models, and data that fits your curve perfectly or the uncertainty on the Y value at X=500 will be so large that you can't draw any conclusions. $\endgroup$ – Joris Meys Apr 25 '12 at 12:31
  • $\begingroup$ Flagged the question for moving it to crossvalidated.com. Some moderator will take care of it shortly, no need to repost the question there. $\endgroup$ – Joris Meys Apr 25 '12 at 12:32
  • $\begingroup$ To be honest I find your comments a bit harsh. I am trying to learn how I can extrapolate such curve with R. This is not a statistics question per-se. And you don't really have to know what the data means. In short I find your actions (closing this topic) completely outside the concept of stackoverlflow. $\endgroup$ – Benoit B. Apr 25 '12 at 14:04
  • 8
    $\begingroup$ Sometimes the truth seems harsh. A local regression is as the name implies ... an estimate determined by the local data. You need theory if you want to extrapolate far beyond your data. lowess plots explicitly eschew theory. $\endgroup$ – DWin Apr 25 '12 at 14:08
4
$\begingroup$

Curiously, I just addressed a similar question here, although that was in the context of a standard linear model, instead of loess. Reading that may give you some of the background ideas. I will take the substance of this question to pertain specifically to loess per se. The theory behind loess is to have a semi-parametric fit that yields a predicted value based only on a few nearby points, weighted by proximity. There is typically a bandwidth argument that gives the range of $x$ values that would be considered 'nearby' (although this may be determined automatically, or set by default). The weights on any existing data point outside this window will be 0. Moreover, whatever the bandwidth is set to, it will certainly not be wider than the width of your data set. Thus, the ideas behind loess absolutely preclude extrapolating a predicted value for $x=500$ from a loess fit based on data that range from [0,100]. Even beyond this however, the predicted value is the one generated when the window is centered on the $x$ value to be used for the prediction--other possible predicted values when that $x$ is within the window, but not in the exact center, are not used or given any weight. You can see how this leads to complications as the window moves towards the ends of the range of $x$; it is often considered that loess is less reliable at the extremes of your existing $x$ range. These facts should make it clear that loess, unfortunately, cannot be used for extrapolating. It is possible that a parametric model could, but my first reaction would be to be very wary even in that case (see my previous answer for a better feel for that). Sorry to be the bearer of bad news...

$\endgroup$
4
$\begingroup$

You can get a better feel for what the loess (lowess) algorythm does by running the loess.demo function from the TeachingDemos package then clicking on the plot to see the weights and window used to predict the curve at that point. This undestanding may help you to see why loess is unlikely to give anything meaningful with that much of an extrapolation.

Theoretically you could set the window wide enough that it could give a prediction at 500, but as the window gets wider the loess fit approaches a straight line and any window wide enough to generate a prediction at 500 would be pretty much the same as fitting a straight line to your data and extrapolating with that.

$\endgroup$
1
$\begingroup$

Extrapolating from a LOWESS or LOESS fit is a very bad idea.

LOWESS works by fitting a weighted linear model to a local subset of the data.

You find the "N" nearest neighbors to your data point. You then fit either a first order or second order polynomial to the data, weighting the regression based on the distance from the data point.

This local regression model is used to estimate a value for your original data point.

You repeat this same process for all of the data points in your sample.

Interpolation is pretty safe (and easy). Connect all your data points with a PCHIP and you're off to the races.

On the other hand, extrapolation is a VERY bad idea. Your local model is based on a very small subset of the data. Your projection is going to be incredibly sensitive to whatever type of assumption you make about your data.

Ultimately, you need to make some kind of assumption how to extrapolate from those last few end points. Consider how radically your estimation might change depending on whether you used linear interpolation as opposed to PCHIP as opposed to a cubic spline.

$\endgroup$
  • $\begingroup$ This would be a great opportunity for some example graphs :D $\endgroup$ – naught101 Dec 11 '18 at 3:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy