I have a data set of 10-dimensional cell measurements for leukemia. The data points are unlabeled and the task is to find the ratio of pathological measurements w.r.t. the rest of the sample. In other words the assumption is that there are two clusters/classes (pathological/healthy).

The problem is that one class vastly outnumbers the other (for a healthy person, the pathological cells would account for ~0.1% of the whole data set [usually upwards of 500k data points], while for a sick person they would be anywhere from 1 to 5%).

I am trying to fit a mixture of gaussians with two components, trained with EM but the model cannot seem to discriminate between healthy and pathological cases, in other words even on training sets that I know to represent a negative sample, the algorithm shows pathological-to-healthy ratios that would normally indicate a positive sample.

The way I see it, there's really only 2 sources for the problem, either the model (GMM) is poor and so cannot represent the data with the desired accuracy or the optimization algorithm (EM) seems to be stuck in poor local optima. I thought I'd check alternative models first, since it seems to me that it would be the easier of the two problems to deal with.

I came across variational autoencoders (https://arxiv.org/abs/1312.6114) and their discrete variants (https://arxiv.org/abs/1609.02200), but I'm wondering whether a variational inference framework would fit this problem. I thought that such algorithms are used when the posterior $p(Z|X)$ (where $Z$ are the latent variables and $X$ are the inputs) is intractable but in my case I can derive an analytical form for it.

So my question is should I still give variational autoencoders a try, given the representational power of neural networks or look at alternatives of making the algorithm more sensitive? What would those be?


  • $\begingroup$ A Bayesian approach with a prior reflecting this asymmetry could help. $\endgroup$ – Xi'an May 7 '18 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.