1
$\begingroup$

One way of defining the variance of a vector is as follows \begin{align*} \text{Var}(g) = \mathbb{E}[ \, \lVert g \rVert_2^2\, ] - \lVert\,\mathbb{E}[ g ] \, \rVert_2^2. \tag{1}\label{1} \end{align*} I have seen this definition used frequently in the context of analyzing SGD, where the convergence rate depends on the variance of the gradient.

Now, using \eqref{1}, the variance can be estimated with monte carlo samples of $g$ which I denote as $g_1, \ldots, g_n$. Specifically, we get the formula \begin{align*} \frac{1}{n}\sum_{i=1}^n \lVert g_i \rVert_2^2 - \left\lVert \frac{1}{n}\sum_{i=1}^n g_i\right\rVert_2^2 \tag{2}\label{2} \end{align*} Compared to the normal definition of sample variance, this definition seems very strange. As far as I know, \eqref{2} has never been used to estimate variance. My question is whether or not \eqref{2} would be a reasonable way to estimate the variance of a vector function. (A more "normal" seeming approach would be to compute the sample variance component-wise, using the typical formula, then taking the norm of that vector).

$\endgroup$
5
  • $\begingroup$ You appear to confuse two distinct things. (1) is not the usual meaning of vector variance: it is merely the variance of the vector's Euclidean length. Formula (2) is a standard estimator for the variance of the length. $\endgroup$
    – whuber
    Commented Jan 12, 2022 at 16:45
  • $\begingroup$ @whuber please give a reference for "(1) is not the usual meaning of vector variance" $\endgroup$
    – Taw
    Commented Jan 12, 2022 at 20:25
  • 1
    $\begingroup$ I believe you are confused because (1) does not seem to be the variance of the length. $\text{Var}(\lVert g \rVert) = \mathbb{E}[\lVert g \rVert^2] - \mathbb{E}[\lVert g \rVert] ^2 $. The first term (the second moment of g) is the same, but the second term is not the same. We can't switch the norm and expectation order (easy to show using counter example). $\endgroup$
    – Taw
    Commented Jan 12, 2022 at 20:39
  • $\begingroup$ Here is an example of a widely cited paper analyzing SGD which uses the definition (1) as the variance of the gradient. arxiv.org/abs/1606.04838 $\endgroup$
    – Taw
    Commented Jan 12, 2022 at 20:42
  • 2
    $\begingroup$ You're partly right. Let's unwrap the notation. Writing $g=(x_1,x_2,\ldots, x_n),$ your formula becomes $$E[||g||^2] - ||E[g]||^2 = E[x_1^2+\cdots+x_n^2] - ||(E[x_1],\ldots,E[x_n])||^2 \\= \sum_i E[x_i^2] - (E[x_1]^2 + \cdots + E[x_n]^2) = \sum_i \operatorname{Var}(x_i).$$ This can be characterized as the trace of the covariance matrix of $g,$ for instance. $\endgroup$
    – whuber
    Commented Jan 12, 2022 at 20:54

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.