5
$\begingroup$

I am learning about Generalized Additive Models (GAMS) and have found in multiple locations a claim that a GAM with one independent variable is identical to a bivariate LOESS regression. Is this claim correct?

I have been unable to obtain the same answer using the loess function in base R and the gam function in the R package mgcv. Here is a link to a similar question from 2009 on a different forum: http://r.789695.n4.nabble.com/Difference-between-gam-and-loess-td881045.html

Below is my R code, which I modified from code provided at the link above. The gam function in R has numerous options. If a GAM with one independent variable is identical to a bivariate LOESS regression could someone provide the option settings needed? I have tried a wide range of values for span in the loess function and am only showing a few.

Thank you for any advice or assistance.

library(mgcv)

set.seed(1234) 

# generate data
x <- sort(runif(100)) 
y <- sin(2*pi*x) + rnorm(10, sd=0.1) 

mgcv.1 <- gam(y ~ s(x), family=gaussian(), weights=NULL, subset=NULL,
          offset=NULL, method = "GCV.Cp",
          optimizer=c("outer", "newton"), control=list(), scale=0,
          select=FALSE, knots=NULL, sp=NULL, min.sp=NULL, H=NULL, gamma=1,
          fit=TRUE, paraPen=NULL, G=NULL, drop.unused.levels=TRUE,
          bs="cr")

base.r <- loess(y ~ x, degree=1, span=0.50) ; summary(base.r$fitted - mgcv.1$fitted)
base.r <- loess(y ~ x, degree=1, span=0.75) ; summary(base.r$fitted - mgcv.1$fitted)
base.r <- loess(y ~ x, degree=2, span=0.50) ; summary(base.r$fitted - mgcv.1$fitted)
base.r <- loess(y ~ x, degree=2, span=0.75) ; summary(base.r$fitted - mgcv.1$fitted)

EDIT

According to lecture notes at the link here:

http://polisci.msu.edu/jacoby/icpsr/regress3/lectures/week4/16.GAM.pdf

the gam function in mgcv only fits smoothing splines, but local polynomial regression could be done in S-PLUS. Perhaps that is why I cannot obtain the same answer with mgcv as I get with the loess function? Perhaps another GAM package in R does allow local polynomial regression?

$\endgroup$
  • $\begingroup$ see my answer below (gam), your edit and the answer seem to be saying pretty much the same thing. $\endgroup$ – Christoph Hanck Jul 9 '15 at 15:49
  • $\begingroup$ @ChristophHanck Thank you, Christoph. I will explore the gam package some more, as well as the car package and possibly others. $\endgroup$ – Mark Miller Jul 9 '15 at 15:53
6
$\begingroup$

Not really a full answer, but too long for a comment: s sets up a spline, whereas loess does a local regression.

In the gam package (maybe mgcv too, not too familiar with that one) you can also feed a local regression, as in

library(gam)

set.seed(1234) 

# generate data
x <- sort(runif(100)) 
y <- sin(2*pi*x) + rnorm(10, sd=0.1) 

gam.1 <- gam(y ~ lo(x))
base.r <- loess(y ~ x) 
summary(base.r$fitted - gam.1$fitted)
plot(base.r$fitted,gam.1$fitted)

That does not produce the same fitted values either, but maybe you can further play around with the settings of lo and loess.

$\endgroup$
5
$\begingroup$

"LOESS" uses local kernel regression but is not a pure local kernel regression.

Local regression for a pre-specified bandwidth or pre-specified set of varying bandwidths can be written as a linear function of the data.

LOESS is however, non-linear, in that it attempts to introduce a degree of "robustification" to outliers (by downweighting large residuals and refitting).

As a result, in the general case, LOESS results won't be exactly reproduced by playing about with the settings for local linear regression, though if the data are sufficiently "nice" there may be a correspondence if the same settings are used -- LOESS also uses a "span" parameter that alters bandwidth to "cover" a given fraction of the data; even without any issues with large residuals, a local regression method would have to adjust bandwidth in the same way to reproduce it.

$\endgroup$
2
$\begingroup$

If your link function is identity (i.e., the error's PDF is Gaussian), a one covariate GAM is nothing else than the smooth version of your scatter plot. And this is generally a locally weighted scatterplot smoother. Read Hastie and Tibshirani 1986, particularly their section 5.2: They fit the GAMs by Fisher local scoring where the weighted least square fit is substituted by the more general (local) smoothing. Although they do not call it a LOESS, they speak about a running lines smoother with weights (dp/dq)^2*V^(-1)), which is basically a local weighted smoother. If your link function is identity the scoring procedure has no iterations and the linear estimator eta reduces to the smooth of the scatter plot.

Why R does not reflect this behaviour I do not know (I know little about R). I guess you have to specify in the LOESS function that your weights are all 1s (otherwise, I think they depend automatically on the distance between the observation points) and/or surely you have to use the same span in both GAM and LOESS.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.