3
$\begingroup$

I was tying to estimate the distribution for a discrete variable. However, suddenly I thought that "Is a simple histogram sufficient? because I have observations for every evaluation point" So, my question is that "Is there any reason we have to use kernel density estimation for a discrete variable?"

$\endgroup$
1

2 Answers 2

3
$\begingroup$
  • Standard kernel density estimators (see the discrete alternative described in the other answer by Ben) assume and estimate a density function of a continuous variable. If your data is discrete, it wouldn't make much sense. You could use it for plotting the data, to have a better feeling about the shape of the distribution, but not much use besides this.
  • Histogram bins the data, so it doesn't enable you to estimate the exact probabilities.
  • You probably would rather want to use empirical probabilities, either cumulative ones, of raw probabilities $\hat p_i = n_i / \sum_j n_j$, where $n_i$ is how many times did you observe $x_i$ in your dataset. If you think about it, using empirical probabilities is equivalent to using a kernel density estimator with bandwidth $h \to 0$, so that the kernel reduces to a Dirac delta function and the kernel density estimator becomes an empirical distribution estimator $\hat f(x) = 1/n \sum_{i=1}^n \mathbf{1}(x=x_i)$.
  • If you would like to smooth the empirical data because it contains random irregularities, you could use Laplace smoothing and assume the "pseudocount" priors $\alpha_i$ (e.g. $\alpha_1 = \alpha_2 = \dots = \alpha_k = 1$) for each of the $n_i$ counts and use the Dirichlet-multinomial model, i.e. the estimated probabilities become

$$ E[p_i] = \frac{n_i + \alpha_i }{\sum_j n_j + \alpha_j } $$

$\endgroup$
4
  • $\begingroup$ Hi Tim. This answer appears to me to be partially wrong, insofar as it denies the existence of KDE methods designed for discrete distributions. (Please have a read of my answer for an example.) I would prefer not to downvote, so would you like to amend? $\endgroup$
    – Ben
    Jun 16, 2021 at 8:04
  • $\begingroup$ @Ben re-worded it. $\endgroup$
    – Tim
    Jun 16, 2021 at 8:16
  • $\begingroup$ Given the existence of these methods in the literature, do you still stand by the assertion that "If your data is discrete, it wouldn't make much sense"? $\endgroup$
    – Ben
    Jun 16, 2021 at 10:47
  • 1
    $\begingroup$ @Ben added "standard" and reference to your answer. For me, it seems to be self-explanatory. I didn't use the discrete KDE before & you already described it, so I don't feel there is a point in adding another bullet for it. People often use things like KDE with Gaussian kernels for estimating the distribution beyond using it for exploratory data analysis and plotting, what is usually wrong. $\endgroup$
    – Tim
    Jun 16, 2021 at 11:54
3
$\begingroup$

Discrete KDEs exist, and are useful when smoothness of the estimator matters

Whilst it may seem counter-intuitive, kernel density estimators actually have been studied in the context of discrete distributions (see e.g., Rajagopalin and Lall 1995), mostly in cases where smoothness of the estimator is important. Discrete KDEs are generally formed by some method that involves smoothing the observed sample data, giving non-zero mass estimates at values that are near to the observed values, even if no sample data occurs at those points.

You mention the histogram as an estimator of a discrete distribution, and this is a bit off. Histograms involve binning (which is itself a crude smoothness technique), but for nonparametric estimation the default estimator is the empirical mass function showing the sample proportions of each observed outcome. Plotting this function is essentially the same as showing a histogram where each "bin" is a single outcome value. The empirical mass function is of course a valid nonparametric estimator for a discrete mass function, and it has several good estimation properties, including the fact that it is a consistent estimator of the true distribution. However, one bad property it has is that it is not generally very smooth for small samples, especially when you have sparse data. Discrete KDEs use smoothing techniques to remedy this deficiency, giving an estimator that is smoother than the empirical mass function, but converging towards this function (which converges to the true distribution) as the sample size increases.

As to when a discrete KDE would be useful, well, there are some statistical problems where you want to estimate a discrete distribution, and smoothness of the estimator is desirable. For example, if you want to estimate the HDR of an unknown discrete distribution then it is generally a good idea to use a smooth KDE estimator for this purpose, rather than the empirical distribution. In this context, smoothness is valuable because it stops the estimated HDR from being too "spiky" in small samples, due to sparseness of the data over the true distribution. There are some other contexts where discrete KDEs are also useful, pretty much all of which involve problems where smoothness of the estimator is valuable for some reason.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.