I've used the following approach in the past to calculate absolution deviation moderately efficiently (note, this a programmers approach, not a statisticians, so indubitably there may be clever tricks like shabbychef's that might be more efficient).
WARNING: This is not an online algorithm. It requires O(n) memory. Furthermore, it has a worst case performance of O(n), for datasets like [1, -2, 4, -8, 16, -32, ...] (i.e. the same as the full recalculation). [1]
However, because it still performs well in many use cases it might be worth posting here. For example, in order to calculate the absolute deviance of 10000 random numbers between -100 and 100 as each item arrives, my algorithm takes less than one second, while the full recalculation takes over 17 seconds (on my machine, will vary per machine and according to input data). You need to maintain the entire vector in memory however, which may be a constraint for some uses. The outline of the algorithm is as follows:
- Instead of having a single vector to store past measurements, use three sorted priority queues (something like a min/max heap). These three lists partition the input into three: items greater than the mean, items less than the mean and items equal to the mean.
- (Almost) every time you add an item the mean changes, so we need to repartition. The crucial thing is the sorted nature of the partitions which means that instead of scanning every item in the list to repartion, we only need to read those items we are moving. While in the worst case this will still require
O(n) move operations, for many use-cases this is not so.
- Using some clever bookkeeping, we can make sure that the deviance is correctly calculated at all times, when repartitioning and when adding new items.
Some sample code, in python, is below. Note that it only allows items to be added to the list, not removed. This could easily be added, but at the time I wrote this I had no need for it. Rather than implement the priority queues myself, I have used the sortedlist from Daniel Stutzbach's excellent blist package, which use B+Trees internally.
Consider this code licensed under the MIT license. It has not been significantly optimised or polished, but has worked for me in the past. New versions will be available here. Let me know if you have any questions, or find any bugs.
from blist import sortedlist
import operator
class deviance_list:
def __init__(self):
self.mean = 0.0
self._old_mean = 0.0
self._sum = 0L
self._n = 0 #n items
# items greater than the mean
self._toplist = sortedlist()
# items less than the mean
self._bottomlist = sortedlist(key = operator.neg)
# Since all items in the "eq list" have the same value (self.mean) we don't need
# to maintain an eq list, only a count
self._eqlistlen = 0
self._top_deviance = 0
self._bottom_deviance = 0
@property
def absolute_deviance(self):
return self._top_deviance + self._bottom_deviance
def append(self, n):
# Update summary stats
self._sum += n
self._n += 1
self._old_mean = self.mean
self.mean = self._sum / float(self._n)
# Move existing things around
going_up = self.mean > self._old_mean
self._rebalance(going_up)
# Add new item to appropriate list
if n > self.mean:
self._toplist.add(n)
self._top_deviance += n - self.mean
elif n == self.mean:
self._eqlistlen += 1
else:
self._bottomlist.add(n)
self._bottom_deviance += self.mean - n
def _move_eqs(self, going_up):
if going_up:
self._bottomlist.update([self._old_mean] * self._eqlistlen)
self._bottom_deviance += (self.mean - self._old_mean) * self._eqlistlen
self._eqlistlen = 0
else:
self._toplist.update([self._old_mean] * self._eqlistlen)
self._top_deviance += (self._old_mean - self.mean) * self._eqlistlen
self._eqlistlen = 0
def _rebalance(self, going_up):
move_count, eq_move_count = 0, 0
if going_up:
# increase the bottom deviance of the items already in the bottomlist
if self.mean != self._old_mean:
self._bottom_deviance += len(self._bottomlist) * (self.mean - self._old_mean)
self._move_eqs(going_up)
# transfer items from top to bottom (or eq) list, and change the deviances
for n in iter(self._toplist):
if n < self.mean:
self._top_deviance -= n - self._old_mean
self._bottom_deviance += (self.mean - n)
# we increment movecount and move them after the list
# has finished iterating so we don't modify the list during iteration
move_count += 1
elif n == self.mean:
self._top_deviance -= n - self._old_mean
self._eqlistlen += 1
eq_move_count += 1
else:
break
for _ in xrange(0, move_count):
self._bottomlist.add(self._toplist.pop(0))
for _ in xrange(0, eq_move_count):
self._toplist.pop(0)
# decrease the top deviance of the items remain in the toplist
self._top_deviance -= len(self._toplist) * (self.mean - self._old_mean)
else:
if self.mean != self._old_mean:
self._top_deviance += len(self._toplist) * (self._old_mean - self.mean)
self._move_eqs(going_up)
for n in iter(self._bottomlist):
if n > self.mean:
self._bottom_deviance -= self._old_mean - n
self._top_deviance += n - self.mean
move_count += 1
elif n == self.mean:
self._bottom_deviance -= self._old_mean - n
self._eqlistlen += 1
eq_move_count += 1
else:
break
for _ in xrange(0, move_count):
self._toplist.add(self._bottomlist.pop(0))
for _ in xrange(0, eq_move_count):
self._bottomlist.pop(0)
# decrease the bottom deviance of the items remain in the bottomlist
self._bottom_deviance -= len(self._bottomlist) * (self._old_mean - self.mean)
if __name__ == "__main__":
import random
dv = deviance_list()
# Test against some random data, and calculate result manually (nb. slowly) to ensure correctness
rands = [random.randint(-100, 100) for _ in range(0, 1000)]
ns = []
for n in rands:
dv.append(n)
ns.append(n)
print("added:%4d, mean:%3.2f, oldmean:%3.2f, mean ad:%3.2f" %
(n, dv.mean, dv._old_mean, dv.absolute_deviance / dv.mean))
assert sum(ns) == dv._sum, "Sums not equal!"
assert len(ns) == dv._n, "Counts not equal!"
m = sum(ns) / float(len(ns))
assert m == dv.mean, "Means not equal!"
real_abs_dev = sum([abs(m - x) for x in ns])
# Due to floating point imprecision, we check if the difference between the
# two ways of calculating the asb. dev. is small rather than checking equality
assert abs(real_abs_dev - dv.absolute_deviance) < 0.01, (
"Absolute deviances not equal. Real:%.2f, calc:%.2f" % (real_abs_dev, dv.absolute_deviance))
[1] If symptoms persist, see your doctor.
