1
$\begingroup$

I'm sure this has been asked, but my stats knowledge is not good enough to know the terms to looks for, so apologies if this is really basic.

I'm collecting a sample of numbers for which I want to have the current average and standard deviation, but I don't want to keep the entire sequence, just recalculate these values for each new sample.

If my sample is currently: 4, 9, 15, 400, 0, 0 then Excel reports the mean (the 'Average' function) as 71.3333, the Std Dev (via 'STDEV.P') as 147.0778 and the variance (VAR.P) as 21631.8888

If I add '9999' to my sample, then mean, stdev & var change to 1489.5714, 3476.6273 and 12086937.39 respectively.

Using this algorithm from https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Two-pass_algorithm

# for a new value newValue, compute the new count, new mean, the new M2.
# mean accumulates the mean of the entire dataset
# M2 aggregates the squared distance from the mean
# count aggregates the number of samples seen so far
def update(existingAggregate, newValue):
    (count, mean, M2) = existingAggregate
    count = count + 1 
    delta = newValue - mean
    mean = mean + delta / count
    delta2 = newValue - mean
    M2 = M2 + delta * delta2

    return existingAggregate

I use 6 as my count (from the original sample), 71.3333 as my mean and 21611.8888 as M2 (the Variance).

I walk through the algorithm using 9999 as my newValue:

count = 6 + 1 = 7
delta = 9999 - 71.3333 = 9927.6667
mean = 71.3333 + 9927.6667/7 = 1489.5714 (Agrees with Excel)
delta2 = 9999 - 1489.5714 = 8509.4286
M2 = 21611.8888 + 9927.6667 * 8509.4286 = 84,500,382.8 (Excel says 12,086,937.39)

Can someone please show me what I've done wrong or misunderstood?

Improve this question
$\endgroup$
11
  • 1
    $\begingroup$ Have you checked on Wikipedia how M2 is supposed to be used? It says that you must divide by count to get the variance. Divide M2 by 7 and you get the correct answer. The sum of squares of differences from the current mean is not equivalent to the variance, that would imply an ever increasing variance with increasing sample size. $\endgroup$
    – andfor
    Commented Feb 2, 2018 at 15:15
  • $\begingroup$ Ah - so M2 aggregates the squared distance from the mean means that is to total SD for all values. I had a hunch that 'aggregates' should be telling me something... $\endgroup$
    – Steve Ives
    Commented Feb 2, 2018 at 15:20
  • 1
    $\begingroup$ We have many posts on so-called "online" procedures for updating various statistics: see stats.stackexchange.com/search?q=online+variance. $\endgroup$
    – whuber ♦
    Commented Feb 2, 2018 at 16:13
  • 1
    $\begingroup$ This meaning of "online" has nothing to do with the Internet or networks: it means precisely what you describe in your question: update the statistics as you go along without having to store all the data. $\endgroup$
    – whuber ♦
    Commented Feb 2, 2018 at 21:44
  • 1
    $\begingroup$ Determining what is significantly "abnormal" is the province of an obscure but important branch of statistics called sequential analysis. The answer is not the same as the one given in books about hypothesis testing. You can find some decent introductory articles on this in places like Wikipedia. Another (much better known) branch of stats that deals with problems like this is statistical quality control. $\endgroup$
    – whuber ♦
    Commented Feb 2, 2018 at 22:00

1 Answer 1

Reset to default
3
$\begingroup$

Have you checked on Wikipedia how M2 is supposed to be used? It says that you must divide by count to get the variance. Divide M2 by 7 and you get the correct answer. The sum of squares of differences from the current mean is not equivalent to the variance, that would imply an ever increasing variance with increasing sample size

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.