1
$\begingroup$

A random data vector containing non-negative values summing to 1 is:

0.02470075959639532
0.031345388546362596
0.03153954918372276
0.036318448810604316
0.06340725977285551
0.11029259333124006
0.11707200006829813
0.11875407465704997
0.15147816276285112
0.3150917632706202

but the kernel density estimator looks like:

enter image description here

why does the x-axis reach past -0.1 even though no data observation is less than 0.0247? I know the line is supposed to be smoothed, but I don't see what prevents kernel density estimation from just truncating at, or dropping off, at or much nearer to 0 on the x-axis like it should, rather than taking a long time to taper. (this is a general question about the underlying maths, but if it means anything, the pandas plot.kde() function in python was used)

$\endgroup$
8
  • $\begingroup$ Why does your random data vector sum up to 1? $\endgroup$
    – gunes
    Aug 16 '20 at 14:03
  • $\begingroup$ simulating probabilities $\endgroup$
    – develarist
    Aug 16 '20 at 14:09
  • $\begingroup$ This complete setup is not correct, just as you wouldn't want to plot a histogram over these values. KDE operates over data points that are iid. $\endgroup$
    – gunes
    Aug 16 '20 at 14:13
  • 1
    $\begingroup$ Your data is not iid, it sums up to 1. It's like a discretised histogram. KDE treats those values as data points and puts some kernel density (here Gaussian) over those. I don't know any general method for your query (don't suppose it exists as well). $\endgroup$
    – gunes
    Aug 16 '20 at 14:18
  • 1
    $\begingroup$ Histograms are always discretized, so ignore that additional word in my explanation. I meant your data is like a normalized histogram (since it sums up to 1). A kde or another histogram over it feels like you're calculating a histogram of histogram values. Also, your data sample is not like a typical random sample, sampled from a distribution $f(x)$ to be approximated. Simply put, I cannot extend your data with new observations, since it wouldn't sum up to 1. $\endgroup$
    – gunes
    Aug 16 '20 at 14:35
1
$\begingroup$

kde in pandas uses Gaussian kernels. Basically, it puts a gaussian over each data point and sums up the densities (with proper normalisation). So, you'll always have tails extending over your data range. Basically, KDE says that although there is no data in this range, there could have been data in another random sample and I'm assigning some small mass to represent that possibility.

$\endgroup$
3
  • $\begingroup$ so if I extract the probability values that the kde generator came up with, that data series would contain some negative values? $\endgroup$
    – develarist
    Aug 16 '20 at 13:27
  • $\begingroup$ what do you mean by extracting probability values? do you mean sampling from approximated distribution? $\endgroup$
    – gunes
    Aug 16 '20 at 13:34
  • $\begingroup$ kde measures the smoothed line right and tells pyplot what to show, so you extract the values from the function itself somehow, even though it possess no such readily-available attribute to call them out, or extract them directly from the generated plot as a data series of same length to the source data (each kde probability sample corresponds to one of the raw source's probability samples) $\endgroup$
    – develarist
    Aug 16 '20 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.