When to use a multivariate normal distribution?

Question

I was reading Bayes Point Machine example from infer.net: http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/Bayes%20Point%20Machine%20tutorial.aspx

The problem is when we have some input features that we want to classify. The classification goal is to learn a weight vector. Output is classified using this condition:

 W x attributes >= 0   (then class = true or 1, otherwise it's false or zero)

My question is: why for the weight vector we should use a VectorGaussian which is a multivariate normal distribution, and why the elements of weight matrix can't be just modeled using independent normal priors?

I implemented it using independent normal priors with mean zero and variance 1, and there is a dramatic change in the results.

Answer 1

The VectorGaussian in the example has an identity covariance matrix, which means the elements of the weight vector already have independent normal priors with mean zero and variance 1. I suspect what you have actually done is replace the definition of w to be of type VariableArray<double> instead of Variable<Vector>. This doesn't change the statistical model, but it does change how inference will be performed. In this case, the inference is less accurate since the approximate posterior is being factorized over all elements of w.

Answer 2

I just borrowed a piece of data from your link

double[] incomes = { 63, 16, 28, 55, 22, 20 };

double[] ages = { 38, 23, 40, 27, 18, 40 };

bool[] willBuy = { true, false, true, true, false, false };

Clearly, the data are collected from 6 subjects, and each (income, age) pair of data are correlated in the sense that the data is from the same subject.

It's a good idea to think your data structure as a spreadsheet, each row representing data from a subject (person), and each column being a feature of interest. Then the data from each row are correlated since they are from that same subject. That is, the features are correlated! It makes no sense at all to assume independence.