2
$\begingroup$

I have recently noticed that the var() function in R returns a different value from the analogous function in the python package numpy. In particular, R's var() function seems to be actually computing what is sometimes called the quasi-variance:

$\text{qVar} = \frac{1}{n-1} \sum_{i=1}^{n} (x_i-\bar x)^2$

Instead of the most common variance definition:

$\text{Var} = \frac{1}{n} \sum_{i=1}^{n} (x_i-\bar x)^2$

As a consequence, the variance of samples [1,-1,1,-1,1,-1,1,-1] according to R (version 3.3.2) is:

sample <- c(1,-1,1,-1,1,-1,1,-1)
var(sample)

While python's numpy uses the standard definition of variance , yielding the expected unit variance:

import numpy as np

sample = [1,-1,1,-1,1,-1,1,-1]
np.var(sample)

I have two questions:

  1. When and why should anybody use the quasi-variance instead of the variance;

  2. Isn't naming a function that computes the quasi-variance as var() super counter-intuitive?

It seems to me like many people might be reporting wrong experimental results because of this issue.

Note: I am aware that both definitions of variance converge for a sufficiently large sample size, but this is not always the case.

$\endgroup$
  • 5
    $\begingroup$ You might be optimistic about your understanding of what is standard and what is not. Open a statistics textbook, and you will very likely find what you call "quasi-variance" as the main feasible estimator of variance. (Interestingly, I hear the term "quasi-variance" for the first time in this context, and I have been around playing with statistics for a while now.) $\endgroup$ – Richard Hardy Mar 30 '17 at 11:27
  • 2
    $\begingroup$ Quoting help("var"): "The denominator n - 1 is used which gives an unbiased estimator of the (co)variance for i.i.d. observations." So, your claim that the documentation (in R; RStudio is just an IDE, i.e., a front-end) says nothing about this is wrong. $\endgroup$ – Roland Mar 30 '17 at 11:56
  • 1
    $\begingroup$ You are right. I did not read the complete entry for the var function. Instead I searched for the term "quasi-variance", which as clarified by the answer was a mistake. I am editing the question to avoid further confusions. $\endgroup$ – Daniel López Mar 30 '17 at 12:02
7
$\begingroup$

What you call "quasi-variance" is simply the sample variance -- which is also the unbiased estimator of population variance based on the sample variance. You actually usually want this formula rather than the population variance (because you rarely compute variance from the whole population).

In statistics, you are usually working with a sample from a population. If you use the population variance ($\frac{1}{n}SS$) on the sample, you will get the correct number for the sample, but you will underestimate the variance in the whole population. Since the population is what you are normally interested in, you should (in most practical cases) use $\frac{1}{n-1}SS$.

Here is a bit of code that demonstrates this:

set.seed(12345)
# generate 100 sample sets each with 10 samples
# each with mean=1 and variance=1
x <- rnorm(1000, mean=1, sd=1)
# calculate 100 sample variances
xvars1 <- sapply(seq(1,1000,by=10), function(i) var(x[i:(i+9)]))
# calculate 100 population variances
xvars2 <- xvars1 * 9 / 10
print(mean(xvars1))
print(mean(xvars2))

Results

> print(mean(xvars1))
[1] 0.9986701
> print(mean(xvars2))
[1] 0.8988031

As you can see, the average population variance calculated from the sample was by 10% lower than the actual variance of the population, from which the samples were drawn. The sample variance was a better estimate.

TBH this is the first time I see the term "quasi-variance". There is nothing quasi about that.

$\endgroup$
  • $\begingroup$ Thanks for your answer, it has been of great help. Just for the record, I read the term "quasi-variance" in the book "Estadística básica con R", which is the textbook for the subject of basic statistics from a Spanish university. Also, could you provide some reference explaining how the sample variance $\frac{1}{n-1}SS$ is a better estimator of the population variance? $\endgroup$ – Daniel López Mar 30 '17 at 11:49
  • 1
    $\begingroup$ It appears that it might be a term frequently used in Spanish, then. I have not seen that in Polish, German on English text books. $\endgroup$ – January Mar 30 '17 at 11:53
  • 2
    $\begingroup$ I found this article from Mathworld to be very informative regarding the topic of my question: mathworld.wolfram.com/SampleVariance.html $\endgroup$ – Daniel López Mar 30 '17 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.