Suppose I want to learn a joint distribution $p(x_1, \ldots, x_n)$ and have a collection of samples $x^k_1, \ldots, x^k_n$ for each $k$. Assume some values $x^k_i$ are unknown, so the samples are incomplete. What are some ways to learn the distribution?

I would be interested in an answer for a general distribution, but would also be interested if there is a simpler method for a Bayesian network.

  • 1
    $\begingroup$ What do I know about how the unknown entries of the $k^{th}$ observation are chosen? If for example each of the $n$ entries of sample $k$ is hidden with probability $q$, then I think it should be possible to reduce the problem to ordinary density estimation. $\endgroup$
    – Dapz
    Dec 12, 2012 at 20:50
  • $\begingroup$ I see how it would work, but the unknown entries will be determined by human choice. Lets assume we had a complete sample, then somebody just erased some of the numbers and asked us how well we can learn from what is remaining. $\endgroup$
    – Peteris
    Dec 12, 2012 at 22:20
  • 1
    $\begingroup$ Sequential imputation is one technique used in Bayesian inference with missing data. But like @Dapz says, it depends on why the data was missing(e.g. can you assume that all entries were equally likely to have come up missing? or, does your measuring instrument tend to fail when numbers are too large?) $\endgroup$
    – jerad
    Dec 13, 2012 at 0:59

1 Answer 1


This problem can be solved by a Gibbs sampler for data that are missing complete at random (MCAR) or missing at random (MAR). A very flexible method is fully conditional specification (FCS). For each k, specify the conditional distribution x_k given the other variables. Fit the regression of the observed x_k on the other variables, and in fill in each missing data by an imputation drawn from the fitted conditional distribution. Then go on to the next k, and iterate a few times over the entire dataset. There are some subtleties involved in drawing the missing data, and if you want correct statistical inferences, you need to do multiple imputation. But the basic idea is very simple, and works wonderfully well. See http://www.stefvanbuuren.nl/publications/FCS%20in%20multivariate%20imputation%20-%20JSCS%202006.pdf for more information.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.