12
$\begingroup$

When visualising one-dimensional data it's common to use the Kernel Density Estimation technique to account for improperly chosen bin widths.

When my one-dimensional dataset has measurement uncertainties, is there a standard way to incorporate this information?

For example (and forgive me if my understanding is naïve) KDE convolves a Gaussian profile with the delta functions of the observations. This Gaussian kernel is shared between each location, but the Gaussian $\sigma$ parameter could be varied to match the measurement uncertainties. Is there a standard way of performing this? I am hoping to reflect uncertain values with wide kernels.

I've implemented this simply in Python, but I do not know of a standard method or function to perform this. Are there any problems in this technique? I do note that it gives some strange looking graphs! For example

KDE comparison

In this case the low values have larger uncertainties so tend to provide wide flat kernels, whereas the KDE over-weights the low (and uncertain) values.

$\endgroup$
  • $\begingroup$ Are you saying the red curves are the variable-width gaussians and the green curve is their sum? (That does not look plausible from these graphs.) $\endgroup$ – whuber Feb 28 '14 at 17:17
  • $\begingroup$ do you know what's measurement error for each observation? $\endgroup$ – Aksakal Feb 28 '14 at 17:22
  • $\begingroup$ @whuber the red curves are the variable width gaussians and the blue curve is their sum. The green curve is the KDE with a constant width, sorry for the confusion $\endgroup$ – Simon Walker Feb 28 '14 at 17:22
  • $\begingroup$ @Aksakal yes, each measurement has a different uncertainty $\endgroup$ – Simon Walker Feb 28 '14 at 17:22
  • $\begingroup$ A side-issue, but it's not a definition of kernel density estimation that you use Gaussian kernels. You can use any kernel you like integrating to 1, although some kernels are more sensible or useful than others.... $\endgroup$ – Nick Cox Feb 28 '14 at 19:07
6
$\begingroup$

It makes sense to vary the widths, but not necessarily to match the kernel width to the uncertainty.

Consider the purpose of the bandwidth when dealing with random variables for which the observations have essentially no uncertainty (i.e. where you can observe them close enough to exactly) - even so, the kde won't use zero bandwidth, because the bandwidth relates to the variability in the distribution, rather than the uncertainty in the observation (i.e. 'between-observation' variation, not 'within-observation' uncertainty).

What you have is essentially additional source of variation (over the 'no observation-uncertainty' case) that's different for every observation.

So as a first step, I'd say "what's the smallest bandwidth I'd use if the data had 0 uncertainty?" and then make a new bandwidth which is the square root of the sum of the squares of that bandwidth and the $\sigma_i$ you'd have used for the observation uncertainty.

An alternative way to look at the problem would be to treat each observation as a little kernel (as you did, which will represent where the observation might have been), but convolve the usual (kde-) kernel (usually fixed-width, but doesn't have to be) with the observation-uncertainty kernel and then do a combined density estimate. (I believe that's actually the same outcome as what I suggested above.)

$\endgroup$
2
$\begingroup$

I would apply the variable bandwidth kernel density estimator, e.g. Local bandwidth selectors for deconvolution kernel density estimation paper attempts to build the adaptive window KDE when the measurement error distribution is known. You stated that you know the error variance, so this approach should be applicable in your case. Here's another paper on similar approach with a contaminated sample: BOOTSTRAP BANDWIDTH SELECTION IN KERNEL DENSITY ESTIMATION FROM A CONTAMINATED SAMPLE

$\endgroup$
0
$\begingroup$

You may wish to consult chapter 6 in "Multivariate Density Estimation: Theory, Practice, and Visualization " by David W. Scott, 1992, Wiley.

For the univariate case (pp 130-131), he derives the normal reference rule for bandwidth selection: $$h = (4/3)^{1/5}\sigma n^{1/5} \qquad (6.17)$$ where $\sigma$ is the variance along your dimension, $n$ is the amount of data and $h$ is the bandwidth (you used $\sigma$ in your question, so don't confuse it in my notation).

The general KDE notation that he uses is: $$ \hat{f}(x) = \frac{1}{nh} \sum_{i=1}^n K\left(\frac{x-x_i}{h}\right)$$ where $K(\cdot)$ is the Kernel function.

$\endgroup$
0
$\begingroup$

Actually, I think the method you proposed is called Probability Density Plot (PDP) as used in Geo-science widely, see a paper here: https://www.sciencedirect.com/science/article/pii/S0009254112001878

However, there are drawbacks as mentioned in the paper above. Such as if the measured errors are small, there will be spikes in the PDF you get in the end. But one can also smooth the PDP just like the way of KDE, just like what @Glen_b♦ has mentioned

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.