1
$\begingroup$

I can't understand the difference between the possible specifications of the family option in the loess command in R. This is what I found in the help of the loess command:

For the default family, fitting is by (weighted) least squares. For family="symmetric" a few iterations of an M-estimation procedure with Tukey's biweight are used. Be aware that as the initial value is the least-squares fit, this need not be a very resistant fit.

Is it a matter of robustness?

About the degree: can I use the same loess command with degree = 0 to obtain the k-nearest neighbors algorithm for regression? Or is it necessary to specify something else (e.g. uniform kernel, weights, ...)?

$\endgroup$
2
$\begingroup$

Loess iteratively estimates the smoothing function. Part of the iteration involves a regression procedure to estimate the parameters of a smoothing spline as a predictive function. The option "degree" is the polynomial degree of the smoothing spline. As noted in the help file, family has to do with how the polynomial is estimated. With gaussian, you are essentially performing OLS, which hopefully is straightforward.

The M-estimator is a more general root-finding procedure. Note with M-estimation, you also get the OLS model if you use the normal score equations as the $\rho$ function and solve them analytically. Tukey's biweight function is locally linear but with tails that taper down to 0 reducing the influence of outliers.

Essentially yes, you can say this is procedure which is "robust to outliers". But that also means it excludes high influence, high leverage points even when they're the right value. On the topic of outliers, basic stat education says very little except that one "looks for outliers" in an analysis, and indeed one should. Where we've gone terribly wrong is inferring, therefore, that they should be excluded and the data omitting these values should be reported as final with minimal description of the methods. Consider however:

  1. Loess is concerned with a mean response over time. For a skewed error distribution, excluding outliers will lead to biased, inconsistent estimates of the mean
  2. Confidence and prediction intervals are underestimated when excluding outliers, since in replications of the study, one does not guarantee the same outlying results will necessarily be excluded
  3. Excluding outliers outright as an automated routine yields little in the manner of sensitivity analysis. Loess does not report which values are effectively excluded according to the biweight function.

So the parameters of the spline that are estimated provide a solution to the system of equations:

$$ \theta: \sum_{i=1}^n \psi (\hat{y} -y) = 0$$

The function

$\psi(x)= x(1-(x^2)/(c^2))^2$ for $|x|<c$; $0$ for $|x|>c$

Where $\theta$ is the vector of parameters to the smoothing spline used to predict $y$, ($\hat{y})$.

https://mathworld.wolfram.com/TukeysBiweight.html

$\endgroup$
4
  • $\begingroup$ Thank you a lot. Let me check if I understood the main point: if I specify family = gaussian the procedure won't be robust towards outliers. The opposite if family = symmetric is specified (with all the related observations you did). I didn't understand if you answered also the question about knn in the first part, sorry. $\endgroup$ – FranzGoogle Apr 8 at 19:55
  • $\begingroup$ Yes. Someone can correct me if I'm wrong, but KNN is an unsupervised learning technique whereas LOESS is supervised. They aren't related at all. The closest analogue as a smoothing technique would be a boxcar filter, but still it's a regression routine that fits a line rather than consider random variables in two-space. $\endgroup$ – AdamO Apr 8 at 20:04
  • $\begingroup$ @AdamO kNN is supervised; you may be thinking of k-means? I think loess with degree 0 is a version of kNN, where predictions are weighted by distance? $\endgroup$ – Ben Reiniger 2 days ago
  • $\begingroup$ @BenReiniger Oh, yes, that's true. KNN is supervised, I'm pretty sure about it. Thanks for the review. $\endgroup$ – FranzGoogle 2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.