I have a naive question.

I have list a

a = [1, 2, 3, 6] .

mymean = sum(a)/len(a)

To calculate the std I use:

math.sqrt(sum([(i-mymean)**2 for i in a])/len(a) = 1.87

But I found this formula which is also right but I do not understand:

math.sqrt(sum([i**2 for i in a])/len(a) - (sum(a)/len(a))**2)

Can you please help understand the second formula?

  • $\begingroup$ What is len(a)? Is it the range of the values? $\endgroup$ – Michael R. Chernick Feb 1 '19 at 20:41
  • $\begingroup$ in python len(a) means the number of element in a list. $\endgroup$ – Medhat Helmy Feb 1 '19 at 22:10
  • $\begingroup$ Okay I get it now. $\endgroup$ – Michael R. Chernick Feb 1 '19 at 23:16
  • $\begingroup$ @Medhat Do NOT use that second formula for computer calculation. In fairly common situations it leads to catastrophic cancellation (i.e. it can be disastrously inaccurate). $\endgroup$ – Glen_b -Reinstate Monica Feb 2 '19 at 2:27
  • 1
    $\begingroup$ Algebraic equivalence does not mean "equally suitable for calculation"! ... If you need a fast (but pretty stable) single pass variance (and hence, standard deviation) calculation, better algorithms are already on site. E.g. there's one here. As an alternative, you can reduce the problem substantially by subtracting a good guess at the mean from each data value (even just subtracting the first observation from each data point before you use the naive formula would improve things). $\endgroup$ – Glen_b -Reinstate Monica Feb 2 '19 at 3:39

The 2 formulas are mathematically equivalent:

$\frac{\sum{(x_i - \mu)^2}}{n} =$

$\frac{\sum{(x_i^2 - 2 x_i \mu + \mu^2)}}{n} =$

$\frac{\sum{x_i^2}}{n} - \frac{2 \mu \sum{x_i}}{n} + \frac{n \mu^2}{n} =$

$\frac{\sum{x_i^2}}{n} - 2 \mu^2 + \mu^2 =$

$\frac{\sum{x_i^2}}{n} - \mu^2$

The first form requires you to loop through the data 2 times, once to compute the mean, then a second time to compute the variance (the square of the standard deviation). The second form can loop through the data 1 time calculating the sum of the values and the sum of the squares of the values, then combining them. The second is preferred when you only want to go through the data once (can give speed advantages for some big data cases), but the first is often less affected by rounding error, so both are still used.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.