Is this controversial that in PCA, we want the variance as large as possible, while in bayesian, a large variance means a low precision? I am wondering about the variance. We say the variance is inverse proportional  to precision, then we also say the variance is proportional to the information, which means that a large variance means you have a lot of information. In my mind, this is a bit controversial. I am confused about that whether a large variance is good or not? or do we need a trade-off criteria?
 A: Say we're measuring some fixed physical quantity (e.g. the mass of an object). But, the measurement is noisy, meaning it's affected by varying processes other than the signal of interest (e.g. noise in the electronics of the weighing scale). In this case, the measurement will be different each time we weigh the object, even though the the true weight is fixed. As you mention, the precision of the measurement is reduced as the variance increases.
On the other hand, consider a method like PCA, where we want to find directions of high variance in the data (and discard directions with low variance). In this case, the intuition is that the high variance directions are more 'relevant'. This intuition works in some cases but not others. A case where it works: We have a multidimensional signal that varies along a small set of directions. The signal is corrupted by white noise that's spread out across all dimensions (unlike the signal). The noise magnitude is lower than the signal magnitude. In this case, discarding low variance directions would reduce the noise but keep the signal.
Or, say we're interested in the relationship between two variables $X$ and $Y$. In this case, we're interested in the extent to which $X$ and $Y$ change together. If $X$ and $Y$ have larger covariance relative to their individual variances, we could say that $X$ contains more information about $Y$ (and vice versa). Note that the reverse is not necessarily true if they're nonlinearly related. In the general case, see mutual information.
Variance has some connections to the information theoretic concept of entropy, which measures how 'spread out' a distribution is. In this case, you could think of them both as being measures of uncertainty. For unimodal distributions, the entropy scales with the variance (but you can break this relationship in other cases). Another way to interpret entropy: Say we're drawing values from some distribution, and we want a way to represent each of the values. That is, to encode them (e.g. as a bit string). Entropy is the average length (in bits) you'd need to perfectly encode the values, using the shortest possible code. Very loosely, this says that more information is required to describe things that are more uncertain.
A: A large variance in and of itself is neither good nor bad. It depends what that variance refers to, is it the variability of the data, the prior, or the posterior? The concept of variance and other measures of variability and the words "information" and "precision" have different meanings depending upon the context. 
From one point of view, large variance does not equal large information - it is the decrease of that large variance that gives information. Information is gained when there is a decrease in variance. So if that large variance were decreased a lot, then there would be a lot of information.  Suppose you have a model (assume it is normal), when you add data the posterior variance will decrease. So information has been gained by addition of the data, or in other words the uncertainty has decreased. Often it is said that more uncertainty requires more information. What this means is that more (useful) data is required, such that the addition of more data will result in a decrease in posterior variance which results in more information, and hence, more precision (less variability). 
Precision and information are closely related concepts to do with the variability. Precision is about "non-variability" - variability is the degree of imprecision. Information is the decrease in variability (as measured by variance). To relate information and precision together, we could say that information is proportional to the decrease in the imprecision, or conversely, information is proportional to the increase in precision. 
