6
$\begingroup$

Variance measures the amount of information in the data set. How?

Mathematically, the variance is the measure of the variability in the data. So, how this can be explained as a measure of the amount of information in the data? Could you please explain this with a simple example?

I read the following post, here, which makes me confused about, how the variance measures the information about the data? I thought it is just a measure of how the point is far away from the mean.

$\endgroup$

5 Answers 5

13
$\begingroup$

Variance measures the amount of information in the data set. How?

Information is a slippery concept, so it pays to be a little concrete. So specialize to the case of a regression model. You have some variables $x_1, x_2, \dotsc, x_p$, say, which you wants to use to predict or explain $Y$. Lets say now that all the observations of $x_1$ are equal, so its variance is zero. This variable will not be useful for explaining $Y$. If you want to study, say, the relationship between education level and income, but your sample only includes college graduates without any post-graduate education, how useful will that be? It is in this sense that variance can be a measure of information in the data.

But this will only make sense in relationship to some given model. "Information" in the abstract, without any qualification, does not make sense. To see that more clearly, another example: let’s say your observations are repeated measurements of the same specimen, length, weight, whatever. The ideal case is that all are equal — no measurement error. Variance in this case represents measurement error, and higher variance corresponds to less information!

$\endgroup$
7
$\begingroup$

One way to think about the amount of information in a dataset is to see how spread out the data points are.

For example, if you had a dataset of five identical body weights x = [120, 120, 120, 120, 120], there is very little information here as there is no spread or variability at all - knowing the weight of one person tells us the weight of the others.

Now, the variance tells us how spread out the data are, by taking the average squared deviations from the mean. So the more spread out our data are, the larger the variance. In x above, the variance is 0, which is consistent with the observation that there is very little information in x.

Edit This is an oversimplified and misleading explanation - please refer to kjetil b halvorsen's answer above for a more nuanced and correct explanation.

$\endgroup$
4
  • 1
    $\begingroup$ Thanks. But why the variance? Why not median, mean, or other quantities? Is that because variance is commonly used in finance? So, it is important for financial maker-decision. $\endgroup$
    – Maryam
    Jun 29 at 10:55
  • 2
    $\begingroup$ The mean, median and other similar single-number summaries of your dataset would give you an idea of what the typical individual or observation looks like rather than focusing on how spread out the data are. You might use different measures to summarize the data you've got in a meaningful manner, depending on the shape and distribution of the data at hand. In the data above, you'll notice that the mean = median = mode = 5, which is another hint that there isn't much information in the data as different measures of the typical weight all yield the same result. $\endgroup$ Jun 29 at 11:10
  • 2
    $\begingroup$ kjetil b halvorsen's answer provides a nice counterexample. $\endgroup$ Jun 30 at 9:15
  • 1
    $\begingroup$ Completely agree, thank you Richard - another reminder of the dangers of providing oversimplified examples. $\endgroup$ Jun 30 at 14:37
6
$\begingroup$

This is not about variance but rather about PCA and principal components. PCA searches for orthogonal principal components (a linear combination of variables) which maximize explained variability (or equivalently minimize the squared distance from the points to the line; think of it as $R^2$ - the higher it is the more variability we explained, which means the points are closer to the fitted line).

When we do this we say that the first PC explains most of the data, i.e. it explains most of the variability, i.e. it contains most of the information about the data, because it's what we can explain best, given our data. However, this does not necessarily mean that this information is of any value to us, the model may be learning associations which are perfectly clear to us or which are not of interest.

In conclusion, this is not about variance, but rather about "explained variance", which is distributed among the PC's and which is interpreted (by some people) as "how much information this particular PC holds given all the rest".

$\endgroup$
2
$\begingroup$

Say you have a dataset about persons with two features: weight and number of eyes. Which of these two features do you think is more informative for describing persons, i.e. given a data point, which feature makes it easier for you to identify the person it represents?

Here, given enough data, weight will have a much greater variance than number of eyes, whose variance will be close to zero. Therefore, we can conclude that weight is more informative than number of eyes.

$\endgroup$
-2
$\begingroup$

Others already answered conveying perfectly some concepts but the major misunderstatement remains to be clarified: variance IS NOT information, is not even vaguely linked to information. In some application as PCA it is a parameter that allows you to evaluate how much information you can extract from your data with that specific tool, but this is not a measure of information of the data at all. I will not go into mathematical explanation because it is probably better to give basic ideas first (and there are very good formal explanation out there): information is the reduction of uncertainty, lack of variance in data does not correlate in any way with the information, it just means that the phenomenon is steady and this is a big amount of information because it reduces the a priori uncertainty about that phenomenon. The key concept is that information depends on the relation between the observer and the data, not on the data itself: if you think to source coding (data compression as an example) small variance means that you can use less symbols to encode the data and so it means less information, but if you think to channel coding (a repetition code for example) in a large set of "1" there could be no "useless" symbol because every symbol carries part of the original information, there are codes way better than the repetion code to increase the amount of information carried by each symbol, still it is the proof that having n+1 equal values conveys more information than n equal values and so even with 0 variance the information increases. Example: you receive a boolean information on a noisy channel with 0.5 cross-channel probability, if you receive "1" you have exactly 0 bit of information because there is no way to know if the sended bit was 0 or 1, if the sender repeats his message 3 times and you receive "111" you know that the probability that the message was 1 is 0.875 and with 0 variance you have a lot of information, passing from no information at all (1) to very good confidence (111).

$\endgroup$
6
  • 2
    $\begingroup$ Re "not even vaguely linked to information." That begs some definition of "information," because the Fisher Information Matrix is intimately related to variances of maximum likelihood estimates. The issue here is that "information" is a heavily overloaded term that has distinct meanings in statistics, engineering, physics, linguistics, philosophy, and even colloquially. $\endgroup$
    – whuber
    Jul 7 at 12:28
  • $\begingroup$ I already gave the definition used in scientific fields (don't know about humanities), that is, by the book, "reduction of uncertainty". Variance and information are two complitely different concepts as are sound and FT, FT is not a sound, it is just one of the many quantitative tools you can use to study a restricted subset of features of a sound. I think the original misunderstanding arose from this: someone said var=info instead of "I treat only the part of information linked to the number of different symbols and their occurrences without regarding their position measured by var" $\endgroup$
    – Boltzmann
    Jul 7 at 21:43
  • 1
    $\begingroup$ Since variance is often characterized as being a part of uncertainty, even such a vague characterization as "reduction of uncertainty" does not rule out the role of variance in understanding information. Your post reads like you have adopted an unnecessarily limited concept of variance (as being purely a descriptive statistic of a stream of bits) and of how it might be used to understand uncertainty and information. The statistical concept of sampling variance would be a more relevant one to discuss. $\endgroup$
    – whuber
    Jul 7 at 21:53
  • $\begingroup$ Please, I know I can not prove it but I also know I am a PhD in EE and I teach courses of Information Theory at University, so believe me, even if you do not still comprehend the concept, this is not "my definition", nor it is vague, this is the only scientific accepted definition of information. I understand that you lack the basics, unfortunately it is difficult to explain everything from 0 in a comment, I recommend the classic book books.google.com/… $\endgroup$
    – Boltzmann
    Jul 8 at 15:51
  • 1
    $\begingroup$ Readers will appreciate seeing that instead of discussing the question you have resorted to an ad hominem attack (based on no evidence or information) and (unverifiable) claims about university degrees. Neither approach carries any weight here on CV. Please see our help center for our expectations about behavior on this site. $\endgroup$
    – whuber
    Jul 8 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.