Consider the situation that we are estimating a $d$-dimensional density (with suitable regularity conditions) using kernel density estimation,

[Method1,conditional density estimation] We can proceed $d$ $1$-dim density estimation sequentially, i.e. $p(X_1),p(X_2\mid X_1),\cdots p(X_d\mid X_{d-1},\cdots X_1)$. In each step we can choose a $1$-dim optimal bandwidth.

[Method2, multivariate density estimation] Alternatively, we can estimate $p(X_1,\cdots X_d)$ directly as a $d$-dim density estimation problem. Thus, we can choose bandwidths for all $d$-coordinates simultaneously.(Of course not necessarily the same bandwidths in different coordinates.)

I have been told that, in kernel density estimation, the optimal bandwidth in each step of Method 1 will be different from optimal bandwidth chosen in each direction when the optimal bandwidth is chosen in a $d$-dim problem in Method 2(Say via Likelihood cross validation).

(1)Can anyone point to me a literature addressing such a problem? I found [Silverman] is not too useful in this question. My primary guess will be something about robustness, but I could be wrong.

(2)What is the major obstacle generalizing the univariate density estimation methods to higher dimension except for sparsity of the data?

(3)What is the well-adopted/popular choice of optimal bandwidth selection in a multivariate density estimation problem?

The second part of this post concerns the comparison between parametric(kernel density estimation) and nonparametric methods of density estimations(For example, Bayesian histogram).

It is generally a difficult problem to estimate the correct dimension of the density [1], but if now we already know the dimension $d$ of the data, then we can use some established methods to proceed our density estimation(Dirichlet or Gaussian process, for instances). And if our only concern now is precision (say MISE) of the estimation towards the density, then

(4)If we turn to Bayesian nonparametric methods of density estimation instead of the kernel density estimation (say Bayesian histogram), which one will perform better, under different cases with or without sparsity.Are there any existing literature that addressed the problem with some simulation studies?


[Silverman]Silverman, Bernard W. Density estimation for statistics and data analysis. Vol. 26. CRC press, 1986.

[1]Is there an accepted method to determine an approximate dimension for manifold learning

I am not an expert in the branch of density estimation so any reference and clarification will be of great help and appreciated. And this is mainly a ref-request as well as request for an overview.

  • $\begingroup$ (1) I'm not sure I understand the question. You are computing "optimal" bandwidths w.r.t different criteria. If you're using different methods,it's only logical to assume they will produce different results. $\endgroup$ – Adrien Mar 15 '17 at 10:14
  • $\begingroup$ (2) I would say the fact that true optimal kernels and bandwidths are not product of 1D kernels and 1D bandwidths, but full d-Dimensional kernels and bandwidths. You can of course restrict yourself to product kernels, but it comes with a price. $\endgroup$ – Adrien Mar 15 '17 at 10:17
  • $\begingroup$ (4) Kernel density estimation is totally nonparametric, why are you saying it's a parametric method? $\endgroup$ – Adrien Mar 15 '17 at 10:20
  • $\begingroup$ @Adrien (1,2)I am asking for the same kernel with different procedures(in different dimensions the optimal bandwidth selection varies, so if we choose the bandwidth stepwisely then the resulting bandwidth may or may not be the same as the optimal bandwidth we choose simultaneously for every dimension). It is not related with product kernel or not; $\endgroup$ – Henry.L Mar 15 '17 at 12:10
  • $\begingroup$ @Adrien (4)Personally I treat every procedure with an explicit basis as parametric(You can also call it nonparametric, but what I have for non parametric in mind is sth like bayesian nonparametrics). Here the basis is kernels and the dimension of the space may or may not be infinite. $\endgroup$ – Henry.L Mar 15 '17 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.