Difference between LOESS and LOWESS

Question

What is the difference between LOESS (locally estimated scatterplot smoothing) and LOWESS (locally weighted scatterplot smoothing)? From Wikipedia I can only see that LOESS is a generalization of LOWESS. Do they have slightly different parameters?

Answer 1

I think it is important to distinguish between methods and their implementations in software. The main difference with respect to the first is that lowess allows only one predictor, whereas loess can be used to smooth multivariate data into a kind of surface. It also gives you confidence intervals. In these senses, loess is a generalization. Both smooth by using tricube weighting around each point, and loess also adds an optional robustification option that re-weights residuals using biweight weighting.

Now for the implementation. In some software, lowess uses a linear polynomial, while loess uses a quadratic polynomial (though you can alter that). The defaults and shortcuts that the algorithms use are often quite different, so that it is hard to get the univariate outputs to match exactly. On the other hand, I am not aware of a case where the choice between the two made a substantive difference.

Answer 2

lowess and loess are algorithms and software programs created by William Cleveland. lowess is for adding a smooth curve to a scatterplot, i.e., for univariate smoothing. loess is for fitting a smooth surface to multivariate data.

Both algorithms use locally-weighted polynomial regression, usually with robustifying iterations. Local regression is a statistical method for fitting smoothing curves without prior assumptions about the shape or form of the curve.

There have been many implementations of Cleveland's approach, but the most well known and most used are probably the implementations in R. The R core stats package contains lowess and loess functions, both based quite closely on Cleveland's original code but with R interfaces.

lowess

lowess was published as a mathematical algorithm by Cleveland (1979) and as a Fortran software program by Cleveland (1981). lowess smoothing become popular when it was included as a function in the New S language in 1988 (Becker et al, 1988) and then later in R. The original Fortran program (dated 1985) is still available from Netlib (https://www.netlib.org/go/lowess). The R stats function is based on a translation of the Fortran program to C and was one of the earliest functions in R.

The lowess function in R is designed for adding smooth curves to plots, so the output is just a list of ordered x coordinates and smoothed y values. This style of output inputs easily into plotting functions in R:

x <- 1:1000
y <- rnorm(1000)
plot(x, y)
l <- lowess(x, y)
lines(l)

The lowess method consists of computing a series of local linear regressions, with each local regression restricted to a window of x-values. Smoothness is achieved by using overlapping windows and by gradually down-weighting points in each regression according to their distance from the anchor point of the window (tri-cube weighting).

To conserve running time and memory, locally-weighted regressions are computed at only a limited number of anchor x-values, usually 100-200 distinct points. Anchor points are at least delta apart, where delta is a parameter input to the algorithm.

The amount of smoothing is determined by the span of the overlapping windows, defined as a proportion of the total number of points. The larger the span, the smoother the curve. The span parameter is called F in the Fortran code and f in the R stats lowess function.

For each anchor point, a weighted linear regression is performed for a window of neighboring points. The neighboring points consist of the smallest set of closest neighbors containing at least span proportion of all points. Each local regression produces a fitted value for that anchor point. Fitted values for other x-values are then obtained by linear interpolation between anchor points.

For the first iteration, the local linear regressions use tri-cube distance weights. Subsequent iterations multiple the distance weights by robustifying weights. Points with residuals greater than 6 times the median absolute residual are assigned weights of zero and otherwise Tukey's biweight function is applied to the residuals to obtain the robust weights. More iterations produce greater robustness. Cleveland originally suggested 2 robustifying iterations. The default in the R stats lowess function is 3.

loess

loess was published as a mathematical model by Cleveland & Devlin (1988), as a Fortran routine by Cleveland & Grosse (1991) and as an S function by Cleveland, Grosse & Shyu (1992). The Netlib repository still contains Cleveland & Grosse's Fortran version from 1990 (http://www.netlib.org/a/loess) and both Fortran and C versions by Cleveland, Grosse & Shyu from 1992 (http://www.netlib.org/a/dloess). See cloess.pdf for extended documentation. Professor Brian Ripley ported Cleveland, Grosse & Shyu's code to R in 1998. The R function was originally part of Prof Ripley's modreg CRAN package and was merged into the stats package in 2003.

The loess function in R is designed to behave like other regression fitting functions in R such as lm and glm. It accepts a model formula:

fit.lo <- loess(y ~ x1 + x2)

and optional prior weights and outputs a fitted model object that can be input to standard R generic functions like summary, fitted, residuals, predict and so on. The output may need a bit of post-processing however before it can be input to plotting functions like lines. Results are output in original data order rather than sorted by any of the x-variates.

In principle, loess is a direct generalization of lowess in that locally weighted univariate regressions are simply replaced by locally weighted multiple regressions. The implementation is more complicated however and it is harder to avoid consuming memory in the multivariate setup.

Comparing lowess to loess

When there is only one x-variable and no prior weights, the R functions lowess and loess are in principle equivalent, so it is possible to make direct comparisons between them. loess has several capabilities that lowess doesn't:

• It accepts a formula specifying the model rather than the x and y matrices

• The model can include multiple predictors, factors and interactions.

• It accepts prior weights.

• It will estimate the "equivalent number of parameters" implied by the fitted curve.

On the other hand, loess is much slower than lowess and sometimes fails when lowess succeeds, so both programs are kept in R.

The default settings of the two programs are very different. Here is an example in which I force lowess and loess to do precisely the same numerical calculation:

> set.seed(2020617)
> x <- 1:1000
> y <- rnorm(1000)
> out.lowess <- lowess(x, y, f=0.3, iter=3, delta=0)
> out.loess <- loess(y~x, span=0.3, degree=1,
+                    family="symmetric", iterations=4,
+                    surface="direct")

The smoothed values from the two functions are the same to machine precision:

> out.lowess$y[1:5]
[1] 0.04257433 0.04239860 0.04222676 0.04205882 0.04189474
> fitted(out.loess)[1:5]
[1] 0.04257433 0.04239860 0.04222676 0.04205882 0.04189474
> all.equal(out.lowess$y, fitted(out.loess))
[1] TRUE

Things to note here:

1. f is the span argument for lowess

2. loess does quadratic (degree=2) local regression by default instead of linear (degree=1)

3. Unless you specify family="symmetric", loess will fit the curve by least squares, i.e., won't do any robustness iterations at all.

4. lowess and loess count iterations differently: iter in lowess means the number of robustness iterations; iterations in loess means the total number of iterations including the least squares fit, i.e., iterations=iter+1

5. I set delta=0 and surface="direct" to force both functions to avoid interpolation by performing a local regression at every unique x-value.

Large data sets

The only aspect in which it is not possible to make loess and lowess agree exactly is in their treatment of large data sets. When x and y are very long, say 10s of thousands of observations, it is impractical and unnecessary to do the local regression calculation exactly, rather it is usual to interpolate between observations that are very close together. This interpolation is controlled by the delta argument to lowess and by the cell and surface arguments to loess.

When there are a large number of observations, lowess groups together those x-values that are closer than a certain distance apart. Although grouping observations based on distance is in principle the best approach, such an approach is impractical in the multivariate x-space that loess is designed to deal with. So loess instead groups observations together based on the number of observations on a cell rather than on distances. Because of this small difference, lowess and loess will almost always give slightly different numerical results for large data sets. lowess is in principle more accurate but the difference is generally small.

Where the difference between lowess and loess becomes significant is in terms of speed and memory usage. lowess remains fast and efficient even for very large datasets of millions of points. On my laptop PC for example lowess takes 3 seconds for a million points with the default span:

> x <- rnorm(1e6)
> y <- rnorm(1e6)
> system.time(l <- lowess(x, y))
   user  system elapsed 
   3.00    0.02    3.01 

and only half a second if the span is reduced:

> system.time(l <- lowess(x, y, f=0.1))
   user  system elapsed 
   0.53    0.00    0.54 

Such large datasets are impractical with loess.

Other robust implementations

The weightedLowess function of the limma Bioconductor package provides an implementation of lowess with the added ability to accept prior weights. weightedLowess provides a novel C translation of the original Fortran code and tries to preserve as many features of the original algorithm design as possible while generalizing the concept of "span" to account for the prior weights.

In principle, the locfit.robust of the locfit CRAN package has the same functionality as weightedLowess but in practice gives different and (to me) less accurate results. Unfortunately, locfit has not been maintained by the original authors for a long time and contains bugs. For example, any attempt to set the iter argument to locfit.robust leads to an error.

The loessFit function of the limma package provides a consistent interface to (wrapper for) lowess, weightedLowess, locfit and loess.

The SAS PROC LOESS provides an implementation of loess with the additional ability to estimate the span by minimizing the AICC criterion.

Other non-robust implementations

The following extend local-regression to more general contexts but omit the robustifying steps of the original algorithm.

The gam CRAN package provides the ability to include lowess curves in generalized linear model fits.

The locfit.raw and locfit functions of the locfit CRAN package extend the ideas of loess to generalized linear models and density estimation (Loader, 1999).

The lowess function of Stata provides an implementation of univariate lowess. It doesn't appear to implement the robustifying iterations and doesn't use interpolation, meaning that a local regression has to be fitted at every x-value.

The lpoly function of Stata provides a flexible approach to multivariate locally-weighted polynomial smoothing with different choices of weights to the loess algorithm.

References

Cleveland, W.S. (1979). Robust Locally Weighted Regression and Smoothing Scatterplots. Journal of the American Statistical Association 74(368), 829-836.

Cleveland, W.S. (1981). LOWESS: A program for smoothing scatterplots by robust locally weighted regression. The American Statistician 35(1), 54.

Becker, R.A., Chambers, J.M. and Wilks, A.R. (1988). The New S Language: a Programming Environment for Data Analysis and Graphics. Wadsworth & Brooks/Cole, Pacific Grove.

Cleveland, W.S., and Devlin, S.J. (1988). Locally-weighted regression: an approach to regression analysis by local fitting. Journal of the American Statistical Association 83(403), 596-610.

Cleveland, W. S. and Grosse, E. H. (1991). Computational methods for local regression. Statistics and Computing 1, 47–62.

Cleveland, W.S., Grosse, E., and Shyu, W.M. (1992). Local regression models. Chapter 8 In: Statistical Models in S, edited by J.M. Chambers and T.J. Hastie, Chapman & Hall/CRC, Boca Raton.

Cleveland W.S., and Loader C. (1996). Smoothing by local regression: principles and methods. In: Härdle W., Schimek M.G. (eds) Statistical Theory and Computational Aspects of Smoothing. Physica-Verlag, Heidelberg.

Loader, C. (1999). Local Regression and Likelihood. Springer, New York.