# Variance of linear combinations of correlated random variables

I understand the proof that $$Var(aX+bY) = a^2Var(X) +b^2Var(Y) + 2abCov(X,Y),$$ but I don't understand how to prove the generalization to arbitrary linear combinations.

Let $$a_i$$ be scalars for $$i\in {1,\dots ,n}$$ so we have a vector $$\underline a$$, and $$\underline X = X_i,\dots ,X_n$$ be a vector of correlated random variables. Then $$Var(a_1X_1 + \dots a_nX_n) = \sum_{i=1}^n a_i^2 \sigma_i^2 + 2 \sum_{i=1}^n \sum_{j>i}^n a_i a_j \text{ Cov}(X_i,X_j)$$ How do we prove this? I imagine there are proofs in the summation notation and in vector notation?

This is just an exercise in applying basic properties of sums, the linearity of expectation, and definitions of variance and covariance

\begin{align} \operatorname{var}\left(\sum_{i=1}^n a_i X_i\right) &= E\left[\left(\sum_{i=1}^n a_i X_i\right)^2\right] - \left(E\left[\sum_{i=1}^n a_i X_i\right]\right)^2 &\scriptstyle{\text{one definition of variance}}\\ &= E\left[\sum_{i=1}^n\sum_{j=1}^n a_i a_j X_iX_j\right] - \left(E\left[\sum_{i=1}^n a_i X_i\right]\right)^2 &\scriptstyle{\text{basic properties of sums}}\\ &= \sum_{i=1}^n\sum_{j=1}^n a_i a_j E[X_iX_j] - \left(\sum_{i=1}^n a_i E[X_i]\right)^2 &\scriptstyle{\text{linearity of expectation}}\\ &= \sum_{i=1}^n\sum_{j=1}^n a_i a_j E[X_iX_j] - \sum_{i=1}^n \sum_{j=1}^n a_ia_j E[X_i]E[X_j] &\scriptstyle{\text{basic properties of sums}}\\ &= \sum_{i=1}^n\sum_{j=1}^n a_i a_j \left(E[X_iX_j] - E[X_i]E[X_j]\right)&\scriptstyle{\text{combine the sums}}\\ &= \sum_{i=1}^n\sum_{j=1}^n a_i a_j\operatorname{cov}(X_i,X_j) &\scriptstyle{\text{apply a definition of covariance}}\\ &= \sum_{i=1}^n a_i^2\operatorname{var}(X_i) + 2\sum_{i=1}^n \sum_{j\colon j > i}^n a_ia_j\operatorname{cov}(X_i,X_j) &\scriptstyle{\text{re-arrange sum}}\\ \end{align} Note that in that last step, we have also identified $\operatorname{cov}(X_i,X_i)$ as the variance $\operatorname{var}(X_i)$.

You can actually do it by recursion without using matrices:

Take the result for $\text{Var}(a_1X_1+Y_1)$ and let $Y_1=a_2X_2+Y_2$.

$\text{Var}(a_1X_1+Y_1)$

$\qquad=a_1^2\text{Var}(X_1)+2a_1\text{Cov}(X_1,Y_1)+\text{Var}(Y_1)$

$\qquad=a_1^2\text{Var}(X_1)+2a_1\text{Cov}(X_1,a_2X_2+Y_2)+\text{Var}(a_2X_2+Y_2)$

$\qquad=a_1^2\text{Var}(X_1)+2a_1a_2\text{Cov}(X_1,X_2)+2a_1\text{Cov}(X_1,Y_2)+\text{Var}(a_2X_2+Y_2)$

Then keep substituting $Y_{i-1}=a_iX_i+Y_i$ and using the same basic results, then at the last step use $Y_{n-1}=a_nX_n$

With vectors (so the result must be scalar):

$\text{Var}(a'\,X)=a'\,\text{Var}(X)\,a$

Or with a matrix (the result will be a variance-covariance matrix):

$\text{Var}(A\,X)=A\,\text{Var}(X)\,A'$

This has the advantage of giving covariances of the various linear combinations whose coefficients are the rows of $A$ on the off-diagonal elements in the result.

Even if you only know the univariate results, you can confirm these by checking element-by-element.

Basically, the proof is the same as the first formula. I will prove it use a very brutal method.

$Var(a_1X_1+...+a_nX_n)=E[(a_1X_1+..a_nX_n)^2]-[E(a_1X_1+...+a_nXn)]^2 =E[(a_1X_1)^2+...+(a_nX_n)^2+2a_1a_2X_1X_2+2a_1a_3X_1X_3+...+2a_1a_nX_1X_n+...+2a_{n-1}a_nX_{n-1}X_n]-[a_1E(X1)+...a_nE(X_n)]^2$

$=a_1^2E(X_1^2)+...+a_n^2E(X_n^2)+2a_1a_2E(X_1X_2)+...+2a_{n-1}a_nE(X_{n-1}X_n)-a_1^2[E(X_1)]^2-...-a_n^2[E(X_n)]^2-2a_1a_2E(X_1)E(X_2)-...-2a_{n-1}a_nE(X_{n-1})E(X_n)$

$=a_1^2E(X_1^2)-a_1^2[E(X_1)]^2+...+a_n^2E(X_n^2)-a_n^2[E(Xn)]^2+2a_1a_2E(X_1X_2)-2a_1a_2E(X_1)E(X_2)+...+2a_{n-1}a_nE(X_{n-1}X_n)-2a_{n-1}a_nE(X_{n-1})E(X_n)$

Next just note:

$a_n^2E(X_n^2)-a_n^2[E(X_n)]^2=a_n\sigma_n^2$

and

$2a_{n-1}a_nE(X_{n-1}X_n)-2a_{n-1}a_nE(X_{n-1})E(X_n)=2a_{n-1}a_nCov(X_{n-1},Xn)$

Just for fun, proof by induction!

Let $P(k)$ be the statement that $Var[\sum_{i=1}^k a_iX_i] = \sum_{i=1}^k a_i^2\sigma_i^2 + 2\sum_{i=1}^k \sum _{j>i}^k a_ia_jCov[X_i, X_j]$

Then $P(2)$ is (trivially) true (you said you're happy with that in the question).

Let's assume P(k) is true. Thus,

$Var[\sum_{i=1}^{k+1} a_iX_i] = Var[\sum_{i=1}^{k} a_iX_i + a_{k+1}X_{k+1}]$

$=Var[\sum_{i=1}^{k} a_iX_i] + Var[a_{k+1}X_{k+1}] + 2 Cov[\sum_{i=1}^{k} a_iX_i,a_{k+1}X_{k+1}]$

$=\sum_{i=1}^k a_i^2\sigma_i^2 + 2\sum_{i=1}^k \sum _{j>i}^k a_ia_jCov[X_i, X_j]+ a_{k+1}^2\sigma_{k+1}^2 + 2Cov[\sum_{i=1}^{k} a_iX_i, a_{k+1}X_{k+1}]$

$=\sum_{i=1}^{k+1} a_i^2\sigma_i^2 + 2\sum_{i=1}^k \sum _{j>i}^k a_ia_jCov[X_i, X_j] + 2\sum_{i=1}^ka_ia_{k+1}Cov[X_i, X_{k+1}]$

$=\sum_{i=1}^{k+1} a_i^2\sigma_i^2 + 2\sum_{i=1}^{k+1} \sum _{j>i}^{k+1} a_ia_jCov[X_i, X_j]$

Thus $P(k+1)$ is true.

So, by induction,

$Var[\sum_{i=1}^n a_iX_i] = \sum_{i=1}^n a_i^2\sigma_i^2 + 2\sum_{i=1}^n \sum _{j>i}^n a_ia_jCov[X_i, X_j]$ for all integer $n \geq 2$.