2
$\begingroup$

I have a list of Double Values(List of distances form a fixed central point), and I repeatedly chuck out some values(based on simple rule of largest value) and see the change in SD.

I would like to calculate it using a rapid/iterative way to calculate SD after removal of a point.

Have read about different incremental approaches to calculate SD/Variance, but all seem to be for updating SD/Variance after addition of a new data value.

http://en.wikipedia.org/wiki/Standard_deviation#Rapid_calculation_methods

https://stackoverflow.com/questions/895929/how-do-i-determine-the-standard-deviation-stddev-of-a-set-of-values

https://stackoverflow.com/questions/1174984/how-to-efficiently-calculate-a-running-standard-deviation

$\endgroup$
1
1
$\begingroup$

You can downdate a variance in exactly the same fashion as you update it; you "undo" the operations that were required to add the observation in the first place (such as subtracting from totals instead of adding).

Take the observation you want to remove. Imagine you just added that observation as the $n$-th observation (of $n$), using one of the algorithms you point to. Now undo those steps you just did, to take it back to the way it was just before it was added (the variance as at $n-1$ observations).

However, an important caveat: updating (done properly) tends to be quite stable, downdating tends to be less so. You can deal with removing the occasional observation (under some restrictions) but if you're doing a lot of adding and removing, loss of numerical accuracy tends to be more of a problem. (Removing largest values may tend to be worse, so additional caution may be required.)

If you have a specific application in mind (such as a windowed standard deviation), it would be best to mention it explicitly.

$\endgroup$
2
  • $\begingroup$ Thnaks, What i am trying to do is trying to filter, and cluster a set of point. Here I have a fixed central point, around which i try and filter out noisy data points, and create a cluster. I do not have a fixed size for a cluster. I keep removing the farthest data point and the drop in the SD gives me a rough idea on when to stop. I generally stop when the drop in SD falls below set precision value(2-3). Hope this gives you an idea of what I am trying to achieve and would like if you could suggest something here. $\endgroup$ – karx Mar 12 '15 at 12:47
  • $\begingroup$ What I'd suggest is that you post a question asking if that's a good way to go about what you're trying to achieve. $\endgroup$ – Glen_b Mar 13 '15 at 3:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.