# Multivariate non-parametric density estimation with many missing values

Apologies in advance if any of my terminology here is wrong, I'm not an expert in statistics. If I've made any mistakes, let me know and I'll correct them.

The task I'm looking for some advice on approaching is parameterless density estimation (particularly kernel density estimation, but if somebody wants to suggest an entirely different approach I'd be interested in that too) in which there are many (100s or 1000s) of variables, but for each datapoint, I only know a small number of them (perhaps less than 100).

The known variables are not flatly distributed, and in fact the value of one variable (known or unknown) may have an effect on how likely another variable is to be observed. Whether or not a particular variable is observed may also have an effect on how likely another variable is to be observed.

Assume here that I have an extremely large number of datapoints. It should also be mentioned that the more often a variable, or combination of variables, is observed, the more important it is for me to be able to predict it. So, for example, if there's a certain value of variable X for which I never observe variable Y, I don't care about being able to estimate Y for that value of X.

Bandwidth selection in this situation is also something I could use some guidance on, for the kernel approach. I'd be happy to hear about either theoretical approaches or R packages.

• Ah, shows how little I know. Thanks for the suggestion of an alternative. By "missing values" I mean that at each datapoint where there are however many features, I only get to observe a small number of those features. An example might be I'm a doctor trying to find the multivariate pdf of, say, 100 different tests I can do on patients, but I'm working from existing data where each patient only got about 10 or tests performed on them. This is probably still relevant to the log-concave function approach you mention. May 12, 2012 at 23:24
• I accidentally cut my previous comment off early, it's edited now. And thanks for the R recommendations. The missing values problem is still something I need advice with, however. May 12, 2012 at 23:29
• Perhaps we should discuss about the goals of your study, in order to check if nonparametric density estimation is what you really need.
– user10525
May 12, 2012 at 23:43
• Procrastinator is correct. We really can't assess from your description why you want kernel density estimation and whether or not you want univariate or multivariate. It sounds like the covariates in a regression problem have missing values and you want to know how to deal with them. Dropping all cases with at least one covariate missing wastes information and can create bias. There is a well-developed theory of missing data that handles these problems I will write a deatiled answer explaining this. When missingness is informative as it sounds like may be your case things get difficult. May 13, 2012 at 0:15
• I don't want to go into a huge amount of detail, so I'll use an example of a hypothetical study which I think has all the same important features as what I'm doing: Consider an online personality test where users are presented with statements, one at a time and in a random order, and choose to rate how true they are on a (continuous) scale from 0 to 100. However they may also skip any questions they don't want to answer. May 13, 2012 at 22:42

For the Naive Bayes approach, you'll need to make the assumption that your feautures are independent, thus redefining the original Bayes Rule: $$$$P_M(t_k|\textbf{x}) = \frac{p(\textbf{x}|t_k) }{p(\textbf{x})} p(t_k) \label{eq:nbmultimcprob}$$$$ where $$\textbf{x}={x_1, x_2,...,x_N}$$ is your vector of features, $$t_k$$ is class $$k$$ (and implies a positive class instance $$t_k=1$$), $$p(\textbf{x}|t_k)$$ is the likelihood, which you can think of as the multivariate KDE across all features, $$p(t_k)$$ is the class prior (often based on class frequency of the training set), and $$p(\textbf{x})=\sum_{k=1}^K p\left(t_{k}\right) \prod_{i=1}^{N} p(x_i|t_k)$$ is the evidence for $$K$$ transient classes. Feature independence converts this to Naive Bayes which is defined as:
$$$$P_M(t_k|x_1, x_2,..., x_N) = \frac{ \prod_{i=1}^{N} p(x_i|t_k) } {p(\textbf{x})} p(t_k)$$$$ where $$p(\textbf{x}|t_k)$$ has been replace by the product of probability densities from each feature's KDE: $$\prod_{i=1}^{N} p(x_i|t_k)$$.