2
$\begingroup$

Is there a clear analytic link from Kernel smoothing, particularly the Nadaraya–Watson estimator, (S-G, H-P, or W-H) smoothing filter?

The "filter" is called by different names in different fields including analytic chemists & engineers, economists, actuaries, and the statisticians (who tend to know several variants).

When I say "kernel method"for smoothing I am referring to an approach like this (link), or possibly this (link). I give an example in code below as well. A Gaussian kernel is a variation on the Normal probability density, which by definition is non-negative. The Epanechnikov and Uniform estimators are defined as piece-wise but share this constraint.

The "filter", like a kernel based smoother, makes a weighted sum of a window of points around the point of interest. Unlike the smoother, the "filter" has negative valued coefficients.

How do I relate the negative weights in the "filter" with a kernel approach? Is there a connection between the two? Are they fundamentally incompatible? How would I make a more robust Gaussian kernel smoother that has negative weights.

What I mean is this:
The coefficients for the 5-element quadratic smoothing filter are "-3, 12,17,12,and -3" with a normalization constant of 35. The absolute values sum to 41. This means ~15% of weight is negative. Is there a derivation from good solid first principles that gives a Normal Kernel smooth with similarly valued coefficients?

For example, let's say that I want estimate y_true from y_meas as given by the following.

set.seed(231529)

#create data
N <- 200
t <- seq(from=1,to=7*pi,length.out = N)
y_true <-  0.23*sin(1.5*t) - 0.5*cos(0.2*t+0.9)

#corrupt with noise
epsilon <- +runif(n = N,min=-0.1,max = 0.1)
y_meas <- y_true + epsilon

#plot it
plot(t,y_meas, xlab = "t", ylab="y")
lines(t,y_true,col="Green",lwd=2)

I can do this with Savitzky-Golay as follows:

#Savitzy-Golay
library(pracma)
y_sg <- savgol(y_meas,   #my data
               fl = 17,
               dorder=0, # snoothing
               forder=3)  # quadratic

lines(t,y_sg,col="Red")

While the Kernel methods are a way to approximate a probability density function there is this: $$ \mu = \frac{\int x\cdot p\left(x\right) } {\int p \left(x\right) } $$ or in a discrete case $$ \mu = \frac{\Sigma x\cdot p\left(x\right) } {\Sigma p\left(x\right) } $$ and so it can make a clean weighted windowed mean for non-uniformly distributed data.

#Kernel
y_k <- numeric(length = length(t) )
for (i in 9:(N-8)){

     xi <- t[(i-8):(i+8)]-t[i]; #domain
     yi <- y_meas[(i-8):(i+8)]; #range

     temp <- density(xi, #the window
                     bw = "sj",
                     kernel = "gaussian")

     #evaluate at our points - 
     w <- interp1(x=temp$x,y=temp$y,xi=xi)

     #normalize
     w <- w/sum(w)

     #find mean
     y_k[i] <- sum(w*yi)
}

lines(t,y_k,col="Blue",lwd=3) 

For the same input data, the SG, using "eyeball norm", seems to do better.
Here is my plot.

enter image description here

The difference between the two is that SG (or such) is robust (negative coefficients) in a way that the classic Gaussian Kernel is not, unless there is a good reason to adjust the weights. Because it has a window, the N-W estimator is tending to regress toward the mean and so it overshoots the valleys and undershoots the peaks.

I appreciate the responses so far, and will go dig through them. Thanks.

$\endgroup$
  • 1
    $\begingroup$ is "(link)" supposed to be a hyperlink? $\endgroup$ – Sycorax says Reinstate Monica Aug 24 '16 at 19:06
  • 1
    $\begingroup$ The Whittaker-Henderson (etc) filters are all (discrete) smoothing splines, which are approached as distinct from kernel methods. However, they are certainly related to them; most obviously both are linear smoothers (i.e. can be written in the form $\hat{y}=Sy$). However, the connections run deeper. In particular you may want to investigate the the connections between various forms of splines and kernel smoothing, including spline kernels. e.g. see here, p39-40. (I think Härdle discusses that connection in Smoothing techniques.) $\endgroup$ – Glen_b -Reinstate Monica Aug 25 '16 at 0:49
  • 1
    $\begingroup$ Indeed, (though less directly related) you might get something out of a look at the discussion in Pearce and Wand "Penalized Splines and Reproducing Kernel Methods" which discusses the connections between those (particular) spline and kernel methods. $\endgroup$ – Glen_b -Reinstate Monica Aug 25 '16 at 0:54
  • $\begingroup$ In statistics does "kernel smoothing", if unqualified, generally refer to 0-th order moving least squares (piecewise constant, but with kernel weight)? $\endgroup$ – GeoMatt22 Aug 25 '16 at 1:44
  • $\begingroup$ @GeoMatt22 often but not always. (If I understand you right that would usually be called Nadaraya-Watson) $\endgroup$ – Glen_b -Reinstate Monica Aug 25 '16 at 3:58
2
$\begingroup$

Savitzky-Golay filters are used to estimate smoothed values for a function and its derivatives, given $(x,f[x])$ pairs. Kernel Density Estimates are used to estimate smoothed values for a probability density function $p[x]$, given a sample of $x$ points.

I am not sure if this answers your question, but moving least squares is a generalization of Savitzky-Golay that uses an arbitrary kernel to weight the points. Savitzky-Golay filtering corresponds to the case of a uniform ("box") kernel, when the function $f[x]$ is sampled on a uniform $x$ grid.

Now in terms of smoothing, that can actually be an issue with higher-order Savitzky-Golay filters (i.e. Runge's phenomenon). In general if some kernel/filter has negative weights, then it can easily result in roughening of the input signal. The simplest example are derivative filters, but even resampling kernels can have these effects. In statistics typically a kernel will be non-negative, so these image processing "interpolation kernels" (e.g. Lanczos) are really a different beast.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.