PCA and proportion of variance explained In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a precise mathematical definition of what "variance explained" means in terms of principal component analysis (PCA)?
For simple linear regression, the r-squared of best fit line is always described as the proportion of the variance explained, but I am not sure what to make of that either. Is proportion of variance here just the extend of deviation of points from the best fit line?
 A: There is a very simple, direct, and precise mathematical answer to the original question. 
The first PC is a linear combination of the original variables $Y_1$, $Y_2$, $\dots$, $Y_p$ that maximizes the total of the $R_i^2$ statistics when predicting the original variables as a regression function of the linear combination. 
Precisely, the coefficients $a_1$, $a_2$, $\dots$, $a_p$ in the first PC, $PC_1 =  a_1Y_1 + a_2Y_2 + \cdots + a_pY_p$, give you the maximum value of $\sum_{i=1}^p R_i^2(Y_i | PC_1)$, where the maximum is taken over all possible linear combinations. 
In this sense, you can interpret the first PC as a maximizer of "variance explained," or more precisely, a maximizer of "total variance explained."
It is "a" maximizer rather than "the" maximizer, because any proportional coefficients $b_i = c\times a_i$, for $c \neq 0$, will give the same maximum. A nice by-product of this result is that the unit length constraint is unnecessary, other than as a device to come up with "a" maximizer.
For references to original literature and extensions, see 
Westfall,P.H., Arias, A.L., and Fulton, L.V. (2017). Teaching Principal Components Using Correlations, Multivariate Behavioral Research, 52, 648-660.  
A: Think about $Y=A+B$ as random variable $Y$ being explained by two new random variables $A$ and $B$. why we do this? Maybe $Y$ is complex but $A$ and $B$ are less complex. Anyhow, the portion of variance of $Y$ is explained by those of $A$ and $B$. $var(Y) = var(A) + var (B) + 2cov(A,B)$. Application of this to the linear regression is simple. Think of $A$ being $b_0+b_1X$ and $B$ is $e$, then $Y=b_0+b_1X+e$. Portion of variance in $Y$ is explained by the regression line, $b_0+b_1X$.
We use "proportion of variance" term because we want to quantify how much regression line is useful to predict (or model) $Y$.
A: @ttnphns has provided a good answer, perhaps I can add a few points.  First, I want to point out that there was a relevant question on CV, with a really strong answer—you definitely want to check it out.  In what follows, I will refer to the plots shown in that answer.  
All three plots display the same data.  Notice that there is variability in the data both vertically and horizontally, but we can think of most of the variability as actually being diagonal.  In the third plot, that long black diagonal line is the first eigenvector (or the first principle component), and the length of that principle component (the spread of the data along that line--not actually the length of the line itself, which is just drawn on the plot) is the first eigenvalue--it's the amount of variance accounted for by the first principle component.  If you were to sum that length with the length of the second principle component (which is the width of the spread of the data orthogonally out from that diagonal line), and then divided either of the eigenvalues by that total, you would get the percent of the variance accounted for by the corresponding principle component.  
On the other hand, to understand the percent of the variance accounted for in regression, you can look at the top plot.  In that case, the red line is the regression line, or the set of the predicted values from the model.  The variance explained can be understood as the ratio of the vertical spread of the regression line (i.e., from the lowest point on the line to the highest point on the line) to the vertical spread of the data (i.e., from the lowest data point to the highest data point).  Of course, that's only a loose idea, because literally those are ranges, not variances, but that should help you get the point.  
Be sure to read the question.  And, although I referred to the top answer, several of the answers given are excellent.  It's worth your time to read them all.  
