# A measure of overall variance from multivariate Gaussian

I am performing some regression task, where I try to discover the underlying multivariate Gaussians from a set of $n$, $p$-dimensional vectors. For example, given a split of the set into $S_i$ and $S_j$ I will calculate the sample means and covariance matrices (${\sum}_{i,j}$)and decide which is the best split based on the information gain(defined by the entropy ($log(det({\sum}_{i,j}))$). And then we recurse on the subsets $S_i$ and $S_j$.

I am trying to define some stopping criteria for this algorithm, which basically should be that when the variance of the distribution is small enough (not sure how to define this threshold) stop to avoid over-fitting to the training data.

So, my questions are:

1) How can I get a measure of overall variance, is it just $trace({\sum})$?

2) How can I choose a suitable threshold?

Thanks

• You can try Frobenius norm of a matrix, which is basically $trace(\Sigma^T * \Sigma)$. For choosing the threshold, you can add a Regularization term in the information gain expression. Mar 15, 2013 at 19:44

Just like the univariate variance is the average squared distance to the mean, $trace(\hat{\bf{\Sigma}})$ is the average squared distance to the centroid: With $\dot{\bf{X}}$ as the matrix of the centered variables, $\hat{\bf{\Sigma}} = \frac{1}{n} \dot{\bf{X}}' \dot{\bf{X}}$ where $\dot{\bf{X}}' \dot{\bf{X}}$ is the matrix of dot products of the columns of $\dot{\bf{X}}$. Its diagonal elements are $\dot{\bf{X}}_{\cdot i}' \dot{\bf{X}}_{\cdot i} = (\bf{X}_{\cdot i} - \overline{X}_{\cdot i})' (\bf{X}_{\cdot i} - \overline{X}_{\cdot i})$, i.e., the squared distance of variable $i$ to its mean. As such, $trace(\hat{\bf{\Sigma}})$ is a natural generalization of the univariate variance.
A second generalization is $det(\hat{\bf{\Sigma}})$: This is a measure for the volume of the ellipsoid that characterizes the distribution. More precisely, $|det(\hat{\bf{\Sigma}})|$ is the factor by which the volume of the unit cube changes after applying the linear transformation $\hat{\bf{\Sigma}}$. (explanation). Here is an illustration for the matrix $\left(\begin{smallmatrix}1 & -.5\\ .5 & .5\end{smallmatrix}\right)$ with determinant $0.75$ (left: before, right: after transformation):