After going through some slightly terse mathematics, I think I have a slight intuition of kernel density estimation. But I am also aware that estimating multivariate density for more than three variables might not be a good idea, in terms of the statistical properties of its estimators.

So, in what sorts of situations should I want to estimate, say, bivariate density using non-parametric methods? Is it worth enough to start worrying about estimating it for more than two variables?

If you can point to some useful links regarding application of estimation of multivariate density, that'd be great.


One typical case for the application of density estimation is novelty detection, a.k.a. outlier detection, where the idea is that you only (or mostly) have data of one type, but you are interested in very rare, qualitative distinct data, that deviates significantly from those common cases.

Examples are fraud detection, detection of failures in systems, and so on. These are situations where it is very hard and/or expensive to gather data of the sort you are interested in. These rare cases, i.e. cases with low probability of occurring.

Most of the times you are not interested on estimating accurately the exact distribution, but on the relative odds (how likely is a given sample to be an actual outlier vs. not being one).

There are dozens of tutorials and reviews on the topic. This one might be a good one to start with.

EDIT: for some people seems odd using density estimation for outlier detection. Let us first agree on one thing: when somebody fits a mixture model to his data, he is actually performing density estimation. A mixture model represents a distribution of probability.

kNN and GMM are actually related: they are two methods of estimating such a density of probability. This is the underlying idea for many approaches in novelty detection. For example, this one based on kNNs, this other one based on Parzen windows (which stress this very idea at the beginning of the paper), and many others.

It seems to me (but it is just my personal perception) that most if not all work on this idea. How else would you express the idea of an anomalous/rare event?

  • $\begingroup$ The note set you outlined (section 6, "density based approach") outlines some very esoteric (far from mean-stream and quiet developed literature on the subject) approaches to outlier detection. Surely, more common applications must exist. $\endgroup$ – user603 Feb 3 '14 at 20:06
  • 2
    $\begingroup$ Sorry, I do not understand your comment. Two very basic examples would be kNN and GMM. These two methods provides estimates of the density of probability, and can be used for such cases. $\endgroup$ – jpmuc Feb 3 '14 at 20:11
  • $\begingroup$ Thanks. what is GMM? I don't think kNN is a mean-stream approach to outlier detection. Can you refer to a recent textbook on robust statistics where it's used in that context? (I looked at the papers in the slide set you pointed to that pertain to outlier detection seem to either be conference procedings or old books) $\endgroup$ – user603 Feb 3 '14 at 20:16
  • $\begingroup$ GMM = gaussian mixture model. In the slides they refer to scores based on kNNs. I personally have used SVMs for novely detection. Regretfully I cannot recommend you a concrete textbook. Maybe these notes (stats.ox.ac.uk/pub/StatMeth/Robust.pdf) are enough. $\endgroup$ – jpmuc Feb 3 '14 at 20:21
  • 1
    $\begingroup$ I agree strongly with @user603. Density estimation is at first sight a very odd and indirect way of trying to find outliers. Your answer would be enhanced by summarizing how that is applied in practice -- and why you think it works well. $\endgroup$ – Nick Cox Feb 5 '14 at 14:31

I guess that the mean-shift algorithm (http://en.wikipedia.org/wiki/Mean-shift) is a good example for an efficient and suited application of kde. The purpose of this algorithm is to locate the maxima of a density function given data $(x_i)$ sampled from that density function and it is entirely based on a kde modeling: $$ f_h(x) \propto \sum_{x_i} \exp( -(x_{i}-x)^{T}\Sigma^{-1} (x_{i}-x)), $$ where $\Sigma^{-1}$ is a covariance matrix (most of the time estimated). This algorithm is widely used in clustering tasks when the number of components is unknown: each discovered mode is a cluster centroid and the closer a sample to a mode the more likely it belongs to the corresponding cluster (everything being weighted properly by the shape of the reconstructed density). The sample data $x_i$ are typically of dimension larger than one: for example, to perform a 2D color image segmentation, the samples can be 5d for (RComponent, GComponent, BComponent, xPosition, yPosition).


Typically, KDE is touted as an alternative to histograms. The main advantage of KDE over histograms, in this context, is to alleviate the effects of arbitrarily chosen parameters on the visual output of the procedure. In particular (and as illustrated in the link above), KDE does not need the user to specify start and end points.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.