5
$\begingroup$

So I'm attempting to design a graph which shows the density of points in 2D space (i.e. a contour plot) with some meaningful values attached to the contours/levels.

I've used two methods to try to understand KDE output.

Seaborn's kdeplot uses statsmodels KDE PDF to get a 2d array of the probability density function.

Scikit-learn does the same thing (presumably) but outputs the log density. I'm assuming here that the log density refers to the log of the above PDF.

So I believe that if I integrate, over all space, the density (taking exponential of log density) of a KDE built from a sample of points, I get 1.

Log Density plot

However, when plotted, this doesn't really help with the meaning. What is happening at, say, the middle contour level? My current understanding is that $\rm{exp}(-1.92) = 0.15$ and that number is the probability density of that level. Integrating that over the area of the level, $\approx 0.15 \times \pi 0.25^2 = 0.03$ gives me a 3% chance of point being located there.

It has been a while since I've looked into these things so a little explanation would be appreciated.

In the end, all I'm looking for is a meaningful way of displaying density on a 2D plot. I've seen plots displaying contours labelled as normalised density (with 25%, 50%, 75%... contours working out from the middle). This is the sort of thing I'm aiming for but not sure as to the meaning of it

code to reproduce above plot:

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.neighbors.kde import KernelDensity

np.random.seed(1000)
x = np.random.rand(10)
y = np.random.rand(10)
data = np.vstack((x, y)).T

kde = KernelDensity().fit(data)

r = np.linspace(0, 1, 1000)
X, Y = np.meshgrid(r, r)

plot_data = np.vstack((X.ravel(), Y.ravel())).T

log_dens = kde.score_samples(plot_data)

plt.contourf(X, Y, log_dens.reshape(X.shape), cmap='Purples')
plt.colorbar()
plt.plot(x, y, 'b.')
plt.show()
$\endgroup$
1
$\begingroup$

Normalized densities are essentially likelihood ratios, not true probability densities. They will no longer integrate to 1. Also, plotting the density values at 25%,50%,75% of the normalized max are not percentiles. To make this interpretable, you need to de-normalize and plot the level sets corresponding to some set of percentiles.

Note that the KDE using Gaussian densities will likely not be Gaussian.

$\endgroup$
  • $\begingroup$ Ok, so if the normalized densities from the kde are likelihood ratios how do I de-normalize them let alone get the percentiles? $\endgroup$ – Lucidnonsense Oct 18 '15 at 19:31
  • $\begingroup$ @Lucidnonsense Not sure how to do it in python, but mathematically, you can divide the normalized density by its integral over $\mathbb{R}^2$ or you can find the point where the normalized density is 1, then determine its distance from each sample point and re-calculate the true density value at the that point. Then, you multiply all points by the true density of this point. $\endgroup$ – user75138 Oct 18 '15 at 19:37
  • $\begingroup$ Typically KDE is done so that it gives you a true density out, ie integral 1. The two packages mentioned do this, so you don't have to worry about any normalization. $\endgroup$ – Dougal Oct 18 '15 at 19:39
  • $\begingroup$ @Lucidnonsense I am not familiar with these packages, so if, as Dougal said, the densities are already properly normalized to integrate to 1, then you can find the level sets that integrate to the 25, 50, and 75th percentiles, for example. $\endgroup$ – user75138 Oct 18 '15 at 19:47
  • $\begingroup$ @Dougal I looked for talk about normalization in the docs but I didn't find it. Can you point it out to me? Thanks Also, I have no idea about how to start finding the percentile levels... (still looking) $\endgroup$ – Lucidnonsense Oct 18 '15 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.