PCA and proportion of variance explained

Question

In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a precise mathematical definition of what "variance explained" means in terms of principal component analysis (PCA)?

For simple linear regression, the r-squared of best fit line is always described as the proportion of the variance explained, but I am not sure what to make of that either. Is proportion of variance here just the extend of deviation of points from the best fit line?

Answer 1

In case of PCA, "variance" means summative variance or multivariate variability or overall variability or total variability. Below is the covariance matrix of some 3 variables. Their variances are on the diagonal, and the sum of the 3 values (3.448) is the overall variability.

   1.343730519   -.160152268    .186470243 
   -.160152268    .619205620   -.126684273 
    .186470243   -.126684273   1.485549631

Now, PCA replaces original variables with new variables, called principal components, which are orthogonal (i.e. they have zero covariations) and have variances (called eigenvalues) in decreasing order. So, the covariance matrix between the principal components extracted from the above data is this:

   1.651354285    .000000000    .000000000 
    .000000000   1.220288343    .000000000 
    .000000000    .000000000    .576843142

Note that the diagonal sum is still 3.448, which says that all 3 components account for all the multivariate variability. The 1st principal component accounts for or "explains" 1.651/3.448 = 47.9% of the overall variability; the 2nd one explains 1.220/3.448 = 35.4% of it; the 3rd one explains .577/3.448 = 16.7% of it.

So, what do they mean when they say that "PCA maximizes variance" or "PCA explains maximal variance"? That is not, of course, that it finds the largest variance among three values 1.343730519 .619205620 1.485549631, no. PCA finds, in the data space, the dimension (direction) with the largest variance out of the overall variance 1.343730519+.619205620+1.485549631 = 3.448. That largest variance would be 1.651354285. Then it finds the dimension of the second largest variance, orthogonal to the first one, out of the remaining 3.448-1.651354285 overall variance. That 2nd dimension would be 1.220288343 variance. And so on. The last remaining dimension is .576843142 variance. See also "Pt3" here and the great answer here explaining how it done in more detail.

Mathematically, PCA is performed via linear algebra functions called eigen-decomposition or svd-decomposition. These functions will return you all the eigenvalues 1.651354285 1.220288343 .576843142 (and corresponding eigenvectors) at once (see, see).

Answer 2

@ttnphns has provided a good answer, perhaps I can add a few points. First, I want to point out that there was a relevant question on CV, with a really strong answer—you definitely want to check it out. In what follows, I will refer to the plots shown in that answer.

All three plots display the same data. Notice that there is variability in the data both vertically and horizontally, but we can think of most of the variability as actually being diagonal. In the third plot, that long black diagonal line is the first eigenvector (or the first principle component), and the length of that principle component (the spread of the data along that line--not actually the length of the line itself, which is just drawn on the plot) is the first eigenvalue--it's the amount of variance accounted for by the first principle component. If you were to sum that length with the length of the second principle component (which is the width of the spread of the data orthogonally out from that diagonal line), and then divided either of the eigenvalues by that total, you would get the percent of the variance accounted for by the corresponding principle component.

On the other hand, to understand the percent of the variance accounted for in regression, you can look at the top plot. In that case, the red line is the regression line, or the set of the predicted values from the model. The variance explained can be understood as the ratio of the vertical spread of the regression line (i.e., from the lowest point on the line to the highest point on the line) to the vertical spread of the data (i.e., from the lowest data point to the highest data point). Of course, that's only a loose idea, because literally those are ranges, not variances, but that should help you get the point.

Be sure to read the question. And, although I referred to the top answer, several of the answers given are excellent. It's worth your time to read them all.

Answer 3

There is a very simple, direct, and precise mathematical answer to the original question.

The first PC is a linear combination of the original variables $Y_1$, $Y_2$, $\dots$, $Y_p$ that maximizes the total of the $R_i^2$ statistics when predicting the original variables as a regression function of the linear combination.

Precisely, the coefficients $a_1$, $a_2$, $\dots$, $a_p$ in the first PC, $PC_1 = a_1Y_1 + a_2Y_2 + \cdots + a_pY_p$, give you the maximum value of $\sum_{i=1}^p R_i^2(Y_i | PC_1)$, where the maximum is taken over all possible linear combinations.

In this sense, you can interpret the first PC as a maximizer of "variance explained," or more precisely, a maximizer of "total variance explained."

It is "a" maximizer rather than "the" maximizer, because any proportional coefficients $b_i = c\times a_i$, for $c \neq 0$, will give the same maximum. A nice by-product of this result is that the unit length constraint is unnecessary, other than as a device to come up with "a" maximizer.

For references to original literature and extensions, see

Westfall,P.H., Arias, A.L., and Fulton, L.V. (2017). Teaching Principal Components Using Correlations, Multivariate Behavioral Research, 52, 648-660.

Answer 4

Think about $Y=A+B$ as random variable $Y$ being explained by two new random variables $A$ and $B$. why we do this? Maybe $Y$ is complex but $A$ and $B$ are less complex. Anyhow, the portion of variance of $Y$ is explained by those of $A$ and $B$. $var(Y) = var(A) + var (B) + 2cov(A,B)$. Application of this to the linear regression is simple. Think of $A$ being $b_0+b_1X$ and $B$ is $e$, then $Y=b_0+b_1X+e$. Portion of variance in $Y$ is explained by the regression line, $b_0+b_1X$.

We use "proportion of variance" term because we want to quantify how much regression line is useful to predict (or model) $Y$.