What is the difference between Local Linear Regression (LLR) and Locally Estimated Scatterplot Smoothing (LOESS)?

Question

I've looked into nonparametric regression packages in R and Python and came across two estimation methods that are relevant for my problem (i.e. replicating the semiparametric estimation in Carneiro, Pedro, James J. Heckman, and Edward J. Vytlacil. 2011. "Estimating Marginal Returns to Education." American Economic Review).

The two, seemingly related, nonparametric routines are called local linear regression (LLR) and LOESS (or its predecessor LOWESS). I've searched the web trying to find an answer as to if and how the two differ but haven't been successful.

Secondly, do you have any experience with either LLR or LOESS estimation in Python? I have tried the package PyQt-Fit, but haven't been able to retrieve the residuals which I need for the replication of Carneiro et al.'s (2011) semiparametric estimation.

If you are aware of any good Python package to run LLR or LOESS and have an idea of how to obtain the residuals (post-estimation), please let me know.

I'm grateful for any help!

Answer 1

According to Wikipedia. "Local regression" is equivalently called "Local polynomial regression". Now this is confusing since linear regression is used to estimate a polynomial trendline by including the higher order terms as regressors in the model. There is really no such thing as polynomial regression except in the sense of using linear regression to estimate a polynomial trendline. So for all intents and purposes, it seems the unifying class is called local regression.

In the Wiki it's further noted that LOESS and LOWESS are specific implementations of local regression. They use a combination of adaptive splines and weights to consistently estimate the local trendline in a nonparametric fashion. It should be noted that LOESS and LOWESS are so tremendously popular, you don't often hear of other forms of local regression. I'm sure some esoteric research has covered alternatives, but my hunch is that it didn't take with the community. Friedman's "supersmoother" is apparently such an alternative.

• W. S. Cleveland, E. Grosse and W. M. Shyu (1992) Local regression models. Chapter 8 of Statistical Models in S eds J.M. Chambers and T.J. Hastie, Wadsworth & Brooks/Cole.

• http://www.slac.stanford.edu/cgi-wrap/getdoc/slac-pub-3477.pdf

• https://stat.ethz.ch/R-manual/R-devel/library/stats/html/supsmu.html

Answer 2

The pylomo (full disclosure, I am the author) library implements convenient methods for localizing arbitrary models. LOESS is local ordinary-least-squares linear regression, so you can easily roll-your-own LOESS model with this library. Here's an example:

from local_models.local_models import *
import numpy as np
import matplotlib.pyplot as plt

X_train = np.linspace(0,6,100).reshape(-1,1)
y_train = np.sin(X_train) + np.random.normal(loc=0,scale=0.3,size=X_train.shape)
y_train = y_train.flatten()
X_test = np.linspace(-1,7,1000).reshape(-1,1)

kernel = GaussianKernel(bandwidth = 1.)
LOESS = LocalModels(sklearn.linear_model.LinearRegression(), kernel=kernel)
LOESS.fit(X_train,y_train) # This just builds an index and stores x and y

y_pred = LOESS.predict(X_test) # This makes local predictions at these various points

plt.plot(X_test, y_pred)
plt.scatter(X_train, y_train,c='r')
plt.show()

Adjust the bandwidth parameter up to move toward a single straight line (more bias, less variance), and down to simply jump from point to point (more variance, less bias).

Answer 3

Regarding the second part of my question, "how to implement LOESS in Python" (although StackOverflow might be the better place to post this):

Statsmodels has a wonderful package which does exactly what I was looking for (http://www.statsmodels.org/devel/generated/statsmodels.nonparametric.smoothers_lowess.lowess.html)

The lowess function returns the predicted y values. These can be used to obtain the residuals as follows:

residual = y_observed - y_predicted