A measure of overall variance from multivariate Gaussian

Question

I am performing some regression task, where I try to discover the underlying multivariate Gaussians from a set of $n$, $p$-dimensional vectors. For example, given a split of the set into $S_i$ and $S_j$ I will calculate the sample means and covariance matrices (${\sum}_{i,j}$)and decide which is the best split based on the information gain(defined by the entropy ($log(det({\sum}_{i,j}))$). And then we recurse on the subsets $S_i$ and $S_j$.

I am trying to define some stopping criteria for this algorithm, which basically should be that when the variance of the distribution is small enough (not sure how to define this threshold) stop to avoid over-fitting to the training data.

So, my questions are:

1) How can I get a measure of overall variance, is it just $trace({\sum})$?

2) How can I choose a suitable threshold?

Thanks

Answer 1

Just like the univariate variance is the average squared distance to the mean, $trace(\hat{\bf{\Sigma}})$ is the average squared distance to the centroid: With $\dot{\bf{X}}$ as the matrix of the centered variables, $\hat{\bf{\Sigma}} = \frac{1}{n} \dot{\bf{X}}' \dot{\bf{X}}$ where $\dot{\bf{X}}' \dot{\bf{X}}$ is the matrix of dot products of the columns of $\dot{\bf{X}}$. Its diagonal elements are $\dot{\bf{X}}_{\cdot i}' \dot{\bf{X}}_{\cdot i} = (\bf{X}_{\cdot i} - \overline{X}_{\cdot i})' (\bf{X}_{\cdot i} - \overline{X}_{\cdot i})$, i.e., the squared distance of variable $i$ to its mean. As such, $trace(\hat{\bf{\Sigma}})$ is a natural generalization of the univariate variance.

A second generalization is $det(\hat{\bf{\Sigma}})$: This is a measure for the volume of the ellipsoid that characterizes the distribution. More precisely, $|det(\hat{\bf{\Sigma}})|$ is the factor by which the volume of the unit cube changes after applying the linear transformation $\hat{\bf{\Sigma}}$. (explanation). Here is an illustration for the matrix $\left(\begin{smallmatrix}1 & -.5\\ .5 & .5\end{smallmatrix}\right)$ with determinant $0.75$ (left: before, right: after transformation):

I do not have a good answer for your second question. But it seems like the original scales of your variables should matter as they define what variance is "small". It might also be worthwile trying some thresholds and check the stability of the resulting estimates with (bootstrap) crossvalidation.