0
$\begingroup$

I have one dimensional data. All data points are larger than 0. The median and mean are about 10 and 25 respectively. The distribution appears to be lognorm but with really high frequency around the median and fat tail, so lognorm does not fit well. Then I am thinking to use Kernel Density Estimation to describe the data. I tried different ways to find the best bandwidth. (Reference: https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/)

R (reference rules)

bw.SJ(data) 
bw.nrd(data)
bw.nrd0(data)
bw.ucv(data)

All results are too small (smaller than 0.2) and the graph shows too many bumps, which makes it difficult to analyze.

Python sklearn (cross validation)

grid = GridSearchCV(KernelDensity(), {'bandwidth': np.linspace(0.1, 1.0, 30)}, cv=20)

I ended up testing values of bandwidths from 0.01 to 50 and the best one was 20. Since 20 is too large, the graph is almost flat and does not fit the data at all.

Do you have any ideas why these methods do not work well with my data? Could you tell me other methods to find better bandwidths?

$\endgroup$
  • $\begingroup$ Use least squares leave One Out Cross validation $\endgroup$ – Repmat Aug 14 '16 at 10:48
  • $\begingroup$ Thank you for your comment. Could you tell me how to implement it using Python? $\endgroup$ – Nickel Aug 14 '16 at 13:16
  • 1
    $\begingroup$ Sorry I don't know python, but here is something which looks useful jakevdp.github.io/blog/2013/12/01/kernel-density-estimation $\endgroup$ – Repmat Aug 14 '16 at 14:19
  • $\begingroup$ Can you post a link to your data - then things become more concrete ... :) $\endgroup$ – wolfies Aug 14 '16 at 17:12
1
$\begingroup$
  1. Your data may be truly multimodal. The bandwidths that you mention work well asymptotically. If your data is large enough, and does not contain many outliers, then your data may be multimodal.
  2. Your data contain many outliers. This is a big issue with KDE since the bandwidth is sensitive to the presence of outliers. The fitted KDE may be way off in such cases.
  3. If you believe your data is unimodal, you may want to compare the fit of the KDE with that of a log concave estimator. These are implemented in the R packages logcondens and LogConcDEAD.
  4. Try a parametric alternative such a log Student t distribution.
  5. Fit your data in the log scale. This may help you visualise some features more clearly and remove the effect of outliers.
$\endgroup$
  • $\begingroup$ Thank you for your answer! I have about 5000 data points, but that has many outliers as I said fat tail. I believe the true distribution is a normal with mean & median = 2 and noise data occur uniformly (at any values), but there is no way to prove it. $\endgroup$ – Nickel Aug 14 '16 at 13:20
  • $\begingroup$ Do you mean KDE can work well with both unimodal data and multimodal data, but logcondens and LogConcDEAD are applicable only to unimodal data? $\endgroup$ – Nickel Aug 14 '16 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.