Apologies in advance if any of my terminology here is wrong, I'm not an expert in statistics. If I've made any mistakes, let me know and I'll correct them.

The task I'm looking for some advice on approaching is parameterless density estimation (particularly kernel density estimation, but if somebody wants to suggest an entirely different approach I'd be interested in that too) in which there are many (100s or 1000s) of variables, but for each datapoint, I only know a small number of them (perhaps less than 100).

The known variables are not flatly distributed, and in fact the value of one variable (known or unknown) may have an effect on how likely another variable is to be observed. Whether or not a particular variable is observed may also have an effect on how likely another variable is to be observed.

Assume here that I have an extremely large number of datapoints. It should also be mentioned that the more often a variable, or combination of variables, is observed, the more important it is for me to be able to predict it. So, for example, if there's a certain value of variable X for which I never observe variable Y, I don't care about being able to estimate Y for that value of X.

Bandwidth selection in this situation is also something I could use some guidance on, for the kernel approach. I'd be happy to hear about either theoretical approaches or R packages.

  • $\begingroup$ Ah, shows how little I know. Thanks for the suggestion of an alternative. By "missing values" I mean that at each datapoint where there are however many features, I only get to observe a small number of those features. An example might be I'm a doctor trying to find the multivariate pdf of, say, 100 different tests I can do on patients, but I'm working from existing data where each patient only got about 10 or tests performed on them. This is probably still relevant to the log-concave function approach you mention. $\endgroup$ May 12, 2012 at 23:24
  • $\begingroup$ I accidentally cut my previous comment off early, it's edited now. And thanks for the R recommendations. The missing values problem is still something I need advice with, however. $\endgroup$ May 12, 2012 at 23:29
  • 2
    $\begingroup$ Perhaps we should discuss about the goals of your study, in order to check if nonparametric density estimation is what you really need. $\endgroup$
    – user10525
    May 12, 2012 at 23:43
  • $\begingroup$ Procrastinator is correct. We really can't assess from your description why you want kernel density estimation and whether or not you want univariate or multivariate. It sounds like the covariates in a regression problem have missing values and you want to know how to deal with them. Dropping all cases with at least one covariate missing wastes information and can create bias. There is a well-developed theory of missing data that handles these problems I will write a deatiled answer explaining this. When missingness is informative as it sounds like may be your case things get difficult. $\endgroup$ May 13, 2012 at 0:15
  • $\begingroup$ I don't want to go into a huge amount of detail, so I'll use an example of a hypothetical study which I think has all the same important features as what I'm doing: Consider an online personality test where users are presented with statements, one at a time and in a random order, and choose to rate how true they are on a (continuous) scale from 0 to 100. However they may also skip any questions they don't want to answer. $\endgroup$ May 13, 2012 at 22:42

1 Answer 1


In order to fit a multivariate kernel density estimate (KDE) with missing data (as it sounds like you are) you need to impute the missing data. Alternatively, you could try using Naive Bayes, which would allow you to fit a unique kernel density estimate per feature, per class, which would alleviate your missing data issue and not require any imputation. Note, Naive Bayes rests on the assumption that your features are independent (although often times we may make this assumption even when it isn't so and the method still works well).

For the first approach, fitting a multivariate KDE with missing data, there are many imputation approaches available, some of which were designed for this purpose. This well-cited 2014 paper compares a few different approaches: Influence of Missing Values Substitutes on Multivariate Analysis of Metabolomics Data, Gromski et al.

For the Naive Bayes approach, you'll need to make the assumption that your feautures are independent, thus redefining the original Bayes Rule: \begin{equation} P_M(t_k|\textbf{x}) = \frac{p(\textbf{x}|t_k) }{p(\textbf{x})} p(t_k) \label{eq:nbmultimcprob} \end{equation} where $\textbf{x}={x_1, x_2,...,x_N}$ is your vector of features, $t_k$ is class $k$ (and implies a positive class instance $t_k=1$), $p(\textbf{x}|t_k)$ is the likelihood, which you can think of as the multivariate KDE across all features, $p(t_k)$ is the class prior (often based on class frequency of the training set), and $p(\textbf{x})=\sum_{k=1}^K p\left(t_{k}\right) \prod_{i=1}^{N} p(x_i|t_k)$ is the evidence for $K$ transient classes. Feature independence converts this to Naive Bayes which is defined as:

\begin{equation} P_M(t_k|x_1, x_2,..., x_N) = \frac{ \prod_{i=1}^{N} p(x_i|t_k) } {p(\textbf{x})} p(t_k) \end{equation} where $p(\textbf{x}|t_k)$ has been replace by the product of probability densities from each feature's KDE: $\prod_{i=1}^{N} p(x_i|t_k)$.

Lastly, you mention bandwidth-setting in your question. As is mentioned in this 2012 paper: Classification Using Kernel Density Estimates, Ghosh et al., bandwidth is often set uniquely for each class (and in the Naive Bayes setting, perhaps for each feature and each class). Bandwidths are often optimized using a grid search over a range of valid values (which is dictated by the values of the dataset, and whether or not you scale the data).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.