PCA and proportion of variance explained

Question

in general, what is meant when it says that the fraction X of the variance in an analysis like PCA is explained by the first principal component? can someone explain this intuitively but also give a precise mathematical definition of what "variance explained" means in terms of a PCA analysis?

for simple linear regression, the r-squared of best fit line is always described as the proportion of the variance explained, but i am not sure what to make of that either. is proportion of variance here just the extend of deviation of points from the best fit line?

thanks.

Answer 1

In case of PCA, "variance" means summative variance or multivariate variability or overall variability. Below is the covariance matrix of some 3 variables. Their variances are on the diagonal, and the sum of the 3 values (3.448) is the overall variability.

   1.343730519   -.160152268    .186470243 
   -.160152268    .619205620   -.126684273 
    .186470243   -.126684273   1.485549631

Now, PCA replaces original variables with new variables, called principal components, which are orthogonal (i.e. they have zero covariations) and have variances (called eigenvalues) in decreasing order. So, the covariance matrix between the principal components extracted from the above data is this:

   1.651354285    .000000000    .000000000 
    .000000000   1.220288343    .000000000 
    .000000000    .000000000    .576843142

Note that the diagonal sum is still 3.448, which says that all 3 components account for all the multivariate variability. The 1st principal component accounts for or "explains" 1.651/3.448 = 47.9% of the overall variability; the 2nd one explains 1.220/3.448 = 35.4% of it; the 3rd one explains .577/3.448 = 16.7% of it.

Answer 2

@ttnphns has provided a good answer, perhaps I can add a few points. First, I want to point out that there was a relevant question on CV, with a really strong answer--you definitely want to check it out. In what follows, I will refer to the plots shown in that answer.

All three plots display the same data. Notice that there is variability in the data both vertically and horizontally, but we can think of most of the variability as actually being diagonal. In the third plot, that long black diagonal line is the first eigenvector (or the first principle component), and the length of that principle component (the spread of the data along that line--not actually the length of the line itself, which is just drawn on the plot) is the first eigenvalue--it's the amount of variance accounted for by the first principle component. If you were to sum that length with the length of the second principle component (which is the width of the spread of the data orthogonally out from that diagonal line), and then divided either of the eigenvalues by that total, you would get the percent of the variance accounted for by the corresponding principle component.

On the other hand, to understand the percent of the variance accounted for in regression, you can look at the top plot. In that case, the red line is the regression line, or the set of the predicted values from the model. The variance explained can be understood as the ratio of the vertical spread of the regression line (i.e., from the lowest point on the line to the highest point on the line) to the vertical spread of the data (i.e., from the lowest data point to the highest data point). Of course, that's only a loose idea, because literally those are ranges, not variances, but that should help you get the point.

Be sure to read the question. And, although I referred to the top answer, several of the answers given are excellent. It's worth your time to read them all.

Answer 3

Think about Y=A+B as random variable Y being explained by two new random variables A and B. why we do this? Maybe Y is complex but A and B are less complex. Anyhow, the portion of variance of Y is explained by those of A and B. var(Y) = var(A) + var (B) + 2cov(A,B). Application this to the linear regression is simple. Think of A being b0+b1X and B is e, then Y=b0+b1X+e. Portion of variance in Y is explained by the regression line, b0+b1X.

We use "proportion of variance" term because we want to quantify how much regression line is useful to predict (or model) Y.