One of properties of PCA states that the sum of the variances of the principal components is equal to the sum of the variances of the explanatory variables. I wonder how to interpret this as I've always thought that we do not consider $X$'s as random variables. I'm quite new to probability theory and I need to get it straight: are explanatory variables random variables (or do we consider them fixed)? And if we don't consider them random, how is it possible to apply the variance operator to a variable that is not stochastic?
$\begingroup$
$\endgroup$
11
-
$\begingroup$ The claim in your first sentence—"One of properties of PCA states that sum of the variances of the principal components is equal to the sum of the variances of the explanatory variables."—is only true when performing PCA on the covariance matrix $\mathbf{\Sigma}$. However, when performing PCA on the correlation matrix $\mathbf{R}$, the assumption is that each variable contributes precisely 1 unit of variance to total variance. $\endgroup$– AlexisCommented Jun 10, 2014 at 22:05
-
$\begingroup$ I hope it wasn't too misleading. PCA was just an example to display the problem I had trouble with :) $\endgroup$– PasatoCommented Jun 10, 2014 at 22:38
-
3$\begingroup$ PCA is simply an applied mathematical method. It itself doesn't assume that the variables being analyzed are random. PCA is indifferent to words "sample", "population" etc. $\endgroup$– ttnphnsCommented Jun 11, 2014 at 5:38
-
2$\begingroup$ @Alexis, saying that it is false is too strong. When the correlation matrix is being analyzed, in essence the variables have all been standardized. Thus the variables all have variances equal to 1 & "the sum of the variances of the principal components is [still] equal to the sum of the variances of the explanatory variables". The question is still sensible even in the case of the correlation matrix. $\endgroup$– gung - Reinstate MonicaCommented Jun 11, 2014 at 17:20
-
1$\begingroup$ @Alexis, maths operations imply something beyond what they show. (Just like words. If it were not so, no philosophy could be ever build on both.) Computing correlations imply standardizing variables, even though you don't calculate those. $\endgroup$– ttnphnsCommented Jun 11, 2014 at 19:48
|
Show 6 more comments
1 Answer
$\begingroup$
$\endgroup$
2
A variable is anything that does (or even can) take different values. Not all variables are random variables. When we do regression analyses, we consider the explanatory / predictor variables to be fixed, even when we sampled their values. This is because we are interested in understanding the response as a function of the explanatory variables. In another context, and with different goals, we can take the explanatory variables as stochatic, if appropriate. There is an interesting philosophical issue here, but it is a bit moot. Even in laboratory experiments, where all variables are controlled and set a-priori exactly at fixed levels, the explanatory variables certainly do have variances (albeit PCA would be tremendously uninteresting in such a case).
answered Jun 10, 2014 at 21:45
-
$\begingroup$ Thanks gung. By the way - is it mathematically legit to take variance of variable which we do not assume to be random? $\endgroup$– PasatoCommented Jun 10, 2014 at 21:48
-
1$\begingroup$ Variance is simply a property of a set of numbers. The existence of a variance does not presuppose that the variable was stochastic. You can take a variance of a variable fixed at prespecified levels. What you might legitimately conclude from the variance may be another issue, though. $\endgroup$ Commented Jun 10, 2014 at 21:51