I am performing some regression task, where I try to discover the underlying multivariate Gaussians from a set of $n$, $p$-dimensional vectors. For example, given a split of the set into $S_i$ and $S_j$ I will calculate the sample means and covariance matrices (${\sum}_{i,j}$)and decide which is the best split based on the information gain(defined by the entropy ($log(det({\sum}_{i,j}))$). And then we recurse on the subsets $S_i$ and $S_j$.

I am trying to define some stopping criteria for this algorithm, which basically should be that when the variance of the distribution is small enough (not sure how to define this threshold) stop to avoid over-fitting to the training data.

So, my questions are:

1) How can I get a measure of overall variance, is it just $trace({\sum})$?

2) How can I choose a suitable threshold?


  • 1
    $\begingroup$ You can try Frobenius norm of a matrix, which is basically $trace(\Sigma^T * \Sigma)$. For choosing the threshold, you can add a Regularization term in the information gain expression. $\endgroup$
    – steadyfish
    Mar 15 '13 at 19:44

Just like the univariate variance is the average squared distance to the mean, $trace(\hat{\bf{\Sigma}})$ is the average squared distance to the centroid: With $\dot{\bf{X}}$ as the matrix of the centered variables, $\hat{\bf{\Sigma}} = \frac{1}{n} \dot{\bf{X}}' \dot{\bf{X}}$ where $\dot{\bf{X}}' \dot{\bf{X}}$ is the matrix of dot products of the columns of $\dot{\bf{X}}$. Its diagonal elements are $\dot{\bf{X}}_{\cdot i}' \dot{\bf{X}}_{\cdot i} = (\bf{X}_{\cdot i} - \overline{X}_{\cdot i})' (\bf{X}_{\cdot i} - \overline{X}_{\cdot i})$, i.e., the squared distance of variable $i$ to its mean. As such, $trace(\hat{\bf{\Sigma}})$ is a natural generalization of the univariate variance.

A second generalization is $det(\hat{\bf{\Sigma}})$: This is a measure for the volume of the ellipsoid that characterizes the distribution. More precisely, $|det(\hat{\bf{\Sigma}})|$ is the factor by which the volume of the unit cube changes after applying the linear transformation $\hat{\bf{\Sigma}}$. (explanation). Here is an illustration for the matrix $\left(\begin{smallmatrix}1 & -.5\\ .5 & .5\end{smallmatrix}\right)$ with determinant $0.75$ (left: before, right: after transformation):

enter image description here

I do not have a good answer for your second question. But it seems like the original scales of your variables should matter as they define what variance is "small". It might also be worthwile trying some thresholds and check the stability of the resulting estimates with (bootstrap) crossvalidation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.