Generative modeling of a mix of continous and discrete variables

Question

I'm trying to build a generative model to run a Monte Carlo simulation. The existing data consist of a combination of discrete and continuous variables. Suppose I have a number of people...

Age Sex Non-white
21  1   1
35  1   1

I can easily use a mixture of Gaussians to model this data set and just use EM to estimate the mixture coefficients. But I'm not sure if the underlying assumptions are sound when the underlying components of the joint aren't necessarily (or at all) Gaussian.

I know a couple of more complicated methods, such as using Bayesian networks or things coming out of machine learning (e.g. Boltzmann Machines), but I doubt that they'll be useful for my rather small data set.

I am wondering if there's a compact way to build generative models of multivariate, correlated categorical variables. Or is mixture of Gaussians generally sufficient?

Answer 1

Different Item Distributions

You might first revisit the derivation of the EM algorithm you'd use to fit any kind of mixture model.

For the finite mixture you describe, observed variables are conditionally independent given the identity of the mixture component. Conditional independence implies that the conditional distributions of the observed variables enter into the computation of the posterior over mixture components separately. And once you get a expectation for the posterior over mixture component identities you can update the parameters of each of your conditional distributions, just as you otherwise. In your case the updates will look different per component, but the principle is the same as if they had all been the same.

Implementation

The only slightly tiresome thing about your problem is that you might have to write the EM algorithm yourself. But hopefully now that's not too difficult. The absolutely simplest set of assumptions about your example suggests two binomials and a gaussian and you could find the relevant E and M steps for that by combining the appropriate elements from the mixture of binomials and the mixture of gaussians derivations that are available all over the web.

Correlated Categorical Variables

You say you are interested in correlated categorical variables. These may be harder to specify and With a small data set are usually also harder to estimate (although you don't say how small...)

One possibility is to formulate a mixture of regression models to couple them, if that fits your problem, although it certainly wouldn't with the example data you present here. The R package flexmix will do this.