2
$\begingroup$

I am trying to understand LOESS regression. I've read the Wikipedia and Wolfram articles on it, and the R help for the loess() function.

I think I understand now how the local polynomials are derived, each by a weighted regression over a subset of the data points that are all sufficiently close to the point being estimated.

What I have been unable to find any explanation of is how a single smooth (here presumably meaning once or twice differentiable) curve across the whole explanatory variable domain is obtained from that collection of polynomials.

I have entertained various possibilities, such as:

  • the domain is chopped into a number of separated intervals, each containing a single data point, with the relevant local polynomial applying in each such interval and an interpolation method being used to bridge the space between one such interval and the next; or
  • each of the polynomials is applied across the entire domain and the final curve is an average of these; or
  • the local polynomials will naturally merge smoothly into one another in sequence, as a consequence of some property of the estimation procedure that was not highlighted in the above reading

I would be grateful if somebody could explain, or point me to a source that contains an explanation, how a single smooth curve is obtained from the set of multiple local polynomials.

Thank you

Andrew

EDIT: I investigated the suggestion that the curve is piecewise-linear through the fitted points, using the R code below. The differences between the piecewise-linear curve and the loess curve points are of the order of 10^-4 times the fitted values, which is small, but not small enough to be explained as rounding error. I attach a plot of the differences. There's something interesting going on here about how the local polynomials are joined, but I can't find any explanation of what it is.

delta<-0.01
func<-function(x) sin((7/8)*(x*2*pi/delta)) # chose a wiggly function to push the boundaries of the regression
x<-delta*seq(-500,500,1)
y<-func(x)
data<-data.frame(x=x,y=y)
data<-data[order(data$x),]
loess.model<-loess(y~x,data=data,span=0.1,degree=2) # fit the loess model
data$fitted<-loess.model$fitted

x.with.finer.grid<- min(x)+seq(0,10*delta,delta/10)

# make vector of loess predictions at and between fitted points
pred.loess<-predict(loess.model, newdata=x.with.finer.grid)
dev.off()
plot(x=x.with.finer.grid,y=pred.loess,tck=1,xaxp=c(min(x),min(x)+10*delta,10))

# create vector of piecewise linear line connecting fitted points
pred.linear<-approx(x=data$x[1:11],y=data$fitted[1:11],xout=x.with.finer.grid)
dev.off()
plot(x=x.with.finer.grid,y=pred.linear$y,tck=1,xaxp=c(min(x),min(x)+10*delta,10))

dev.off()
diff<-pred.linear$y-pred.loess
plot(diff)
lines(diff)

quickview<-data.frame(loess=pred.loess,linear=pred.linear$y,diff=diff)
quickview

Graph of diffs between loess and piecewise-linear curves

$\endgroup$

1 Answer 1

1
$\begingroup$

The local weighted regression (whether polynomial or linear) for the $i$th data point yields the value $\hat{y}_i$ as a prediction from the local weighted regression at the values of the covariates for $i$th data point. In other words the prediction for current target point is that predicted by the local weighted regression fitted to the data within the window of width $f$ centred on the focal/target observation. This is repeated for all data points, hence we get a vector of predicted values, one for each observation in the original data.

To form the final smooth, these predicted points are joined with linear segments to form a smooth curve.

$\endgroup$
3
  • $\begingroup$ Thank you for this answer. I just wanted to check: if the points are joined by linear segments, the curve will be piecewise linear and hence continuous but only piecewise differentiable. That doesn't accord with the usual understanding of 'smooth' (eg in splining, continuity of second derivative is generally required). In this context, by 'smooth' do they just mean continuous and piecewise differentiable? $\endgroup$ Commented Mar 8, 2017 at 6:48
  • $\begingroup$ You'd need to read Cleveland's papers on Lowess / Loess in more detail to check, but it doesn't use splines so results associated with those don't apply to Loess. It is a smoother, just a particular type of smoother. There are many types of smoother and not all will be continuously differentiable. Even some splines won't have continuous second derivatives. The feature you describe therefore is not a general property of a smoother. $\endgroup$ Commented Mar 11, 2017 at 20:42
  • $\begingroup$ Do you know which passage of Cleveland's paper(s) explain this joining operation between $\hat{y}_i$? I am also trying to understand this step in the algorithm. $\endgroup$
    – Tanguy
    Commented Apr 1, 2018 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.