28
$\begingroup$

I have read (for example, here) that the Epanechnikov kernel is optimal, at least in a theoretical sense, when doing kernel density estimation. If this is true, then why does the Gaussian show up so frequently as the default kernel, or in many cases the only kernel, in density estimation libraries?

$\endgroup$
12
  • 3
    $\begingroup$ Two questions conflated here: why not more commonly used? why is Gaussian often the default/only kernel? It may sound trivial, but the name Epanechnikov may seem hard to spell and pronounce correctly for people not fluent in that language. (I'm not even sure E. was Russian; I've failed to find any biographical details.) Also, if I show (e.g.) a biweight, comment on its bell shape, finite width and behaviour at the edges, that seem easier to sell. Epanechnikov is the default in Stata's kdensity. $\endgroup$
    – Nick Cox
    Jun 2, 2016 at 6:40
  • 4
    $\begingroup$ I would add that this theoretical optimality has little bearing in practice if any. $\endgroup$
    – Xi'an
    Jun 2, 2016 at 7:36
  • 2
    $\begingroup$ It's a familiar name. If it makes sense to use a kernel that doesn't have a finite support, you should prefer it. So far as my experience goes, it doesn't make sense, so the choice appears social, not technical. $\endgroup$
    – Nick Cox
    Jun 2, 2016 at 18:18
  • 2
    $\begingroup$ @NickCox, yes, E was a Russian dude, it's not an abbreviation :) He was enigmatic person, this is all you could ever find about him. I also remember a very useful book someone with his name wrote on programmable calculators, yes, it was a big thing at the time $\endgroup$
    – Aksakal
    Nov 16, 2018 at 21:05
  • 1
    $\begingroup$ @amoeba He worked at Институт радиотехники и электроники Российской Академии Наук им. Котельникова, I bet he did classified research, full name is Епанечников Виктор Александрович $\endgroup$
    – Aksakal
    Nov 19, 2018 at 14:48

2 Answers 2

18
$\begingroup$

The reason why the Epanechnikov kernel isn't universally used for its theoretical optimality may very well be that the Epanechnikov kernel isn't actually theoretically optimal. Tsybakov explicitly criticizes the argument that the Epanechnikov kernel is "theoretically optimal" in pp. 16-19 of Introduction to Nonparametric Estimation (section 1.2.4).

Trying to summarize, under some assumptions on the kernel $K$ and a fixed density $p$ one has that the mean integrated square error is, of the form

$$\frac{1}{nh} \int K^2 (u) du + \frac{h^4}{4}S_K^2 \int (p''(x))^2 dx \,. \tag{1} $$

The main criticism of Tsybakov seems to be minimizing over non-negative kernels, since it's often possible to get better performing estimators, which are even non-negative, without restricting to non-negative kernels.

The first step of the argument for the Epanechnikov kernel begins by minimizing $(1)$ over $h$ and all non-negative kernels (rather than all kernels of a wider class) to get an "optimal" bandwidth for $K$

$$ h^{MISE}(K) = \left( \frac{\int K^2}{nS_K^2 \int (p'')^2} \right)^{1/5}$$

and the "optimal" kernel (Epanechnikov)

$$K^*(u) = \frac{3}{4}(1-u^2)_+ $$

whose mean integrated square error is:

$$h^{MISE}(K^*) = \left( \frac{15}{n \int (p'')^2} \right)^{1/5} \,. $$

These however aren't feasible choices, since they depend on knowledge (via $p''$) of the unknown density $p$ -- therefore they are "oracle" quantities.

A proposition given by Tsybakov implies that the asymptotic MISE for the Epanechnikov oracle is:

$$\lim_{n \to \infty} n^{4/5} \mathbb{E}_p \int (p_n^E (x) - p(x))^2 dx = \frac{3^{4/5}}{5^{1/5}4} \left( \int (p''(x))^2 dx \right)^{1/5} \,. \tag{2} $$

Tsybakov says (2) is often claimed to be the best achievable MISE, but then shows that one can use kernels of order 2 (for which $S_K =0$) to construct kernel estimators, for every $\varepsilon >0$, such that

$$ \limsup_{n \to \infty} n^{4/5} \mathbb{E}_p \int (\hat{p}_n (x) - p(x))^2 dx \le \varepsilon \,. $$

Even though $\hat{p}_n$ isn't necessarily non-negative, one still has the same result for the positive part estimator, $p_n^+ := \max(0, \hat{p}_n)$ (which is guaranteed to be non-negative even if $K$ isn't):

$$ \limsup_{n \to \infty} n^{4/5} \mathbb{E}_p \int (p_n^+ (x) - p(x))^2 dx \le \varepsilon \,. $$

Therefore, for $\varepsilon$ small enough, there exist true estimators which have smaller asymptotic MISE than the Epanechnikov oracle, even using the same assumptions on the unknown density $p$.

In particular, one has as a result that the infimum of the asymptotic MISE for a fixed $p$ over all kernel estimators (or positive parts of kernel estimators) is $0$. So the Epanechnikov oracle is not even close to being optimal, even when compared to true estimators.

The reason why people advanced the argument for the Epanechnikov oracle in the first place is that one often argues that the kernel itself should be non-negative because the density itself is non-negative. But as Tsybakov points out, one doesn't have to assume that the kernel is non-negative in order to get non-negative density estimators, and by allowing other kernels one can non-negative density estimators which (1) aren't oracles and (2) perform arbitrarily better than the Epanechnikov oracle for a fixed $p$. Tsybakov uses this discrepancy to argue that it doesn't make sense to argue for optimality in terms of a fixed $p$, but only for optimality properties which are uniform over a class of densities. He also points out that the argument still works when using the MSE instead of MISE.

EDIT: See also Corollary 1.1. on p.25, where the Epanechnikov kernel is shown to be inadmissible based on another criterion. Tsybakov really seems not to like the Epanechnikov kernel.

$\endgroup$
3
  • 6
    $\begingroup$ +1 for an interesting read, but this does not answer why Gaussian kernel is used more often than Epanechnikov kernel: they are both non-negative. $\endgroup$
    – amoeba
    Nov 16, 2018 at 21:04
  • $\begingroup$ @amoeba That is true. At the very least this answers the question in the title, which is only about the Epanechnikov kernel. (I.e. it addresses the premise for the question and shows that it is false.) $\endgroup$ Nov 16, 2018 at 21:25
  • 5
    $\begingroup$ (+1) One thing to beware with Tsybakov's scheme of taking the positive part of a possibly-negative kernel estimate – which is at least my memory of his suggestion – is that although the resulting density estimator might give better MSE convergence to the true density, the density estimate will in general not be a valid density (since you're cutting off mass, and it no longer integrates to 1). If you actually only care about MSE, it doesn't matter, but sometimes this will be a significant problem. $\endgroup$
    – Danica
    Nov 16, 2018 at 21:33
2
$\begingroup$

The Gaussian kernel is used for example in density estimation through derivatives:

$$\frac{d^if}{dx^i}(x)\approx \frac{1}{bandwidth}\sum_{j=1}^N \frac{d^ik}{dx^i}(X_j,x)$$

This is because the Epanechnikov kernel has 3 derivatives before it's identically zero, unlike the Gaussian which has infinitely many (nonzero) derivatives. See section 2.10 in your link for more examples.

$\endgroup$
5
  • 2
    $\begingroup$ The first derivative of the Epanechnikov (note the second n, by the way) kernel is not continuous where the function crosses the kernel's own bounds; that might be more of an issue. $\endgroup$
    – Glen_b
    Jun 1, 2016 at 22:13
  • 1
    $\begingroup$ @Glen_b: You're probably right, although having 0 derivatives after some $i$ would be silly too. $\endgroup$
    – Alex R.
    Jun 1, 2016 at 22:14
  • 1
    $\begingroup$ @AlexR. While what you say is true, I don't understand how it explains why the Gaussian is so common in ordinary density estimation (as opposed to estimating the derivative of the density). And even when estimating derivatives, section 2.10 suggests that the Gaussian is never the preferred kernel. $\endgroup$ Jun 1, 2016 at 23:43
  • $\begingroup$ @JohnRauser: Keep in mind that you need to use higher order Epanechnikov kernels for optimality. Usually people use a Gaussian because it's just easier to work with and has nicer properties. $\endgroup$
    – Alex R.
    Jun 2, 2016 at 1:14
  • 1
    $\begingroup$ @AlexR I'd quibble on "[u]sually people use a Gaussian"; do you have any systematic data on frequency of use or this is just an impression based on work you see? I see biweights often, but I wouldn't claim more than that. $\endgroup$
    – Nick Cox
    Jun 2, 2016 at 6:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.