49

Parzen window density estimation is another name for kernel density estimation. It is a nonparametric method for estimating continuous density function from the data. Imagine that you have some datapoints $x_1,\dots,x_n$ that come from common unknown, presumably continuous, distribution $f$. You are interested in estimating the distribution given your data. ...


27

Corresponding to any batch of data $X = (x_1, x_2, \ldots, x_n)$ is its "empirical density function" $$f_X(x) = \frac{1}{n}\sum_{i=1}^{n} \delta(x-x_i).$$ Here, $\delta$ is a "generalized function." Despite that name, it isn't a function at all: it's a new mathematical object that can be used only within integrals. Its defining property is that for any ...


22

An alternative is the approach of Kooperberg and colleagues, based on estimating the density using splines to approximate the log-density of the data. I'll show an example using the data from @whuber's answer, which will allow for a comparison of approaches. set.seed(17) x <- rexp(1000) You'll need the logspline package installed for this; install it if ...


21

One solution, borrowed from approaches to edge-weighting of spatial statistics, is to truncate the density on the left at zero but to up-weight the data that are closest to zero. The idea is that each value $x$ is "spread" into a kernel of unit total area centered at $x$; any part of the kernel that would spill over into negative territory is removed and ...


21

A kernel density estimate is a mixture distribution; for every observation, there's a kernel. If the kernel is a scaled density, this leads to a simple algorithm for sampling from the kernel density estimate: repeat nsim times: sample (with replacement) a random observation from the data sample from the kernel, and add the previously sampled random ...


20

You are correct that the area under the curve of a density function represents the probability of getting an x value between a range of x values But remember area is not just height: width is also important. So if you have a spike at 0, if the width is very small (say 0.1) then the height can be quite a bit higher than 1 (up to 10, if the spike is ...


16

One typical case for the application of density estimation is novelty detection, a.k.a. outlier detection, where the idea is that you only (or mostly) have data of one type, but you are interested in very rare, qualitative distinct data, that deviates significantly from those common cases. Examples are fraud detection, detection of failures in systems, and ...


15

The optimal bandwidth for derivative estimation will be different from the bandwidth for density estimation. In general, every feature of a density has its own optimal bandwidth selector. If your objective is to minimize mean integrated squared error (which is the usual criterion) there is nothing subjective about it. It is a matter of deriving the value ...


14

You can think of the Kernel Density Estimation as a smoothed histogram. Histograms are limited by the fact that they are inherently discrete (via bins) and are thus more appropriate for displaying data on discrete variables and can be very sensitive to bin size. What you are actually doing with the Kernel Density Estimation is estimating the probability ...


13

It is a measure of the standard error of the sample mean when there is serial dependence. If $Y_t$ is covariance stationary with $E(Y_t)=\mu$ and $Cov(Y_t,Y_{t-j})=\gamma_j$ (in an iid setting, this quantity would be zero!) such that $\sum_{j=0}^\infty|\gamma_j|<\infty$. Then $$\lim_{T\to\infty}\{Var[\sqrt{T}(\bar{Y}_T- \mu)]\}=\lim_{T\to\infty}\{TE(\bar{...


12

I'm going to provide an (incomplete) answer here in case it helps anyone else out. There are several recent mathematical methods for computing the KDE more efficiently. One is the Fast Gauss Transform, published in several studies including this one. Another is to use a tree-based approach (KD tree or ball tree) to work out which sources contribute to a ...


12

The cosine kernel is not a beta distribution. Note that the following things are all true of the standard cosine density: $f(0)=1$ $f(0.5)=0.5$ The right half of this density is rotationally symmetric about $x=\frac12$: (i.e. considering the other two properties it implies $1-f(x)=f(1-x)$ ) But no beta density on (-1,1) will have all these properties ...


12

There's no need to integrate anything if you know the cdf of the kernel itself. I believe this is straightforward for all the common kernels. Note that a KDE is a mixture density the cdf of a mixture is the mixture of the cdfs. that is, if $\hat{f}(x)=\frac{1}{n}\sum_i f_i(x)$ is your KDE at $x$, then $\hat{F}(x)=\frac{1}{n}\sum_i F_i(x)$. Take a ...


12

A kernel density estimator (KDE) produces a distribution that is a location mixture of the kernel distribution, so to draw a value from the kernel density estimate all you need do is (1) draw a value from the kernel density and then (2) independently select one of the data points at random and add its value to the result of (1). Here is the result of this ...


11

A simple way is to rasterize the domain of integration and compute a discrete approximation to the integral. There are some things to watch out for: Make sure to cover more than the extent of the points: you need to include all locations where the kernel density estimate will have any appreciable values. This means you need to expand the extent of the ...


11

The comments in the code seem to end up defining the two essentially identically (aside a relatively small difference in the constant). Both are of the form $cAn^{-1/5}$, both with what looks like the same $A$ (estimate of scale), and $c$'s very close to 1 (close relative to the typical uncertainty in the estimate of the optimum bandwidth). [The binwdith ...


11

Method 1: Higher-order Pearson systems The Pearson system is, by convention, taken to be the family of solutions $p(x)$ to the differential equation: $$ \frac{d p (x)}{dx} \; = \; -\frac{(a+x) }{c_0 + c_1 x + c_2 x^2} \; p(x) $$ where the four Pearson parameters $(a, c_0, c_1, c_2)$ can be expressed in terms of the first four moments of the population. ...


11

A1. This sounds like a sensible plan to me. Just to mention a couple of points. You'll want to test with different error metrics ($L^p$, K-L divergence, etc.) since methods will perform differently depending on the loss function. Also, you'll want to test for different number of samples. Finally, many density estimation methods perform notoriously badly near ...


10

For the sake of completeness, here's how I ended up doing this in R: # simulate two samples a <- rnorm(100) b <- rnorm(100, 2) # define limits of a common grid, adding a buffer so that tails aren't cut off lower <- min(c(a, b)) - 1 upper <- max(c(a, b)) + 1 # generate kernel densities da <- density(a, from=lower, to=upper) db <- density(...


10

Here's an algorithm to sample from an arbitrary mixture $f(x) = \frac1N \sum_{i=1}^N f_i(x)$: Pick a mixture component $i$ uniformly at random. Sample from $f_i$. It should be clear that this produces an exact sample. A Gaussian kernel density estimate is a mixture $\frac1N \sum_{i=1}^N \mathcal{N}(x; x_i, h^2)$. So you can take a sample of size $N$ by ...


10

If you have a really wide bandwidth, you'll get one peak in your KDE. If you reduce it a bit, its still one peak. Keep reducing it until you get to the switchover point of adding a second peak. That bandwidth is $h(1)$. Now make it smaller still, until you get to the switchover between two peaks and three. That's $h(2)$. And so forth. At any bandwidth ...


10

I hate animations in Web pages, but this question begs for an animated answer: These are KDEs for a set of three values (near -2.5, 0.5, and 2.5). Their bandwidths continually vary, growing from small to large. Watch as three peaks become two and ultimately one. A KDE puts a pile of "probability" at each data point. As the bandwidth widens, the pile "...


9

The area of overlap of two kernel density estimates may be approximated to any desired degree of accuracy. 1) Since the original KDEs have probably been evaluated over some grid, if the grid is the same for both (or can easily be made the same), the exercise could be as easy as simply taking $\min(K_1(x),K_2(x))$ at each point and then using the trapezoidal ...


9

Your description seems to be confusing two different things. Can you show an example of what you're talking about? (It may not be a good idea to use an ordinary kernel density estimate if your random variable is discrete.) You can get negative $x$-values ending up with some positive density from a kernel density estimate, simply because of the way KDEs ...


9

You want to sample posterior using the data and model given. In this case you can: sample from posterior normal distribution with given mean and covariance matrix - use model.predict with full_covariance=True in case; use built-in function model.posterior_samples_f that does the job for you. A sample code is below: import GPy import numpy as np ...


9

You are asking about two things: kernel density estimation and some particular kernel used in kernel density estimation. For the first question you can find some introduction in Can you explain Parzen window (kernel) density estimation in layman's terms? and How to interpret the bandwidth value in a kernel density estimation? threads. As about the ...


8

The AMISE allows one to obtain an expression for the optimal bandwidth for the unknown density $f$. Unfortunately, the expression is in terms of derivatives of $f$. However, it is possible to derive a similar expression giving the optimal bandwidth for a kernel estimate of those derivatives. This is expressed in terms of even higher derivatives of $f$. And ...


8

You could fit a two-component mixture model using http://cran.r-project.org/web/packages/mixtools/index.html. Try using normalmixEM. You could then follow Erich Schubert's suggestions and find the region where Pr[data point was generated from the component with the smaller mean] >= 0.50. Edit: example R code: library(mixtools) simulate <- function(...


8

The answering question is "why do you model your data as a sample from a distribution?" If you want to learn something about the phenomenon behind your data, like when improving a scientific theory or testing a scientific hypothesis, using a non-parametric kernel estimator does not tell you much more than the data istself. While a parameterised model can ...


8

The reason why the Epanechnikov kernel isn't universally used for its theoretical optimality may very well be that the Epanechnikov kernel isn't actually theoretically optimal. Tsybakov explicitly criticizes the argument that the Epanechnikov kernel is "theoretically optimal" in pp. 16-19 of Introduction to Nonparametric Estimation (section 1.2.4). Trying ...


Only top voted, non community-wiki answers of a minimum length are eligible