Why does more variance imply more information?

Question

I just started to read about PCA in machine learning , and got to know that the main goal to determine principal components is to maximize variance so that more information is retained.But, why does more variance imply more information ? According to me , considering something like 'coefficient of unalikeability' would make more sense. For example, consider a data set having 2 features F1 and F2 , where

F1 = (0.1,0.2,0.3,0.4,0.5,0.6) ,variance = 0.03

F2 = (0.1,0.1,0.1,0.9,0.9,0.9) ,variance = 0.16

Now, here according to me F1 has more information than F2 while variance of F2 is more than F1.

Is there a flaw in my understanding of PCA?

Answer 1

When thinking about PCA I visualize taking a picture of a 3d object (projecting the 3d image into 2d). Consider a partially deflated (american) football. From what perspective would the picture show the greatest amount of the football? You would certainly want a full view of the length of the football (1st PC: direction of greatest variance). The second PC would correspond to the width of the football (direction of second greatest variance), and the third would correspond to the depth (which is not much because the football has been flattened.) You would be best able to identify the 3d object in 2d using the view found by PCA: the length and width, disregarding the depth (the dimension with the lowest variance)

Answer 2

Remember PCA is a dimension reduction technique. The idea is to have as much variance explained, will retaining as few principal components as possible.

You cannot randomly choose to keep the second component, but not the first one. Even if the second component explains a higher percentage of the variation - this is due to the eigenvalues.

A nice way to see how many components you should retain, is by using a scree plot.

Answer 3

In the context of PCA the variance is understood as a portion of variance in the original data retained in PCs. Suppose you have the data set with P input series $x_i$. The total variance of the data set is sum of variances of all variables, i.e. $\Omega=\sum_{i=1}^P \operatorname {Var}[x_i]$. After PCA you convert these variables into P factors $f_i$. The sum of variances is retained, of course, $\sum_{i=1}^P \operatorname {Var}[f_i]=\Omega$.

However, when PCA works it will rearrange the variance in such a way that first few factors, say $K\ll P$, will retain most of the total variance $$\frac{\sum_{i=1}^K\operatorname {Var}[f_i]}{\Omega}\sim 1$$ That's what's meant by maximizing variance in the context of PCA.

I guess your confusion comes from the use of the word variance which has other meanings. For instance, large error variance would suggest that there's a lot of noise and amount of information is smaller relative to noise. In this context more variance would not be good for gathering information. In the context of PCA the more variance has a specific meaning reflecting the value of PCA for your data set in terms of retaining information in small number of factors.