Possible Duplicate:
Simple algorithm for online outlier detection of a generic time series

How could I get rid of sparky data in a descrete data set, but in a "smoother out" manner?

Take for instance


There are two sparks, at 20000, but the next one at 600 is also considered a spark.

I've managed to get the very high ones to zero, by

a = 2
b = 5
beta_dist = RealDistribution('beta', [a, b])
f(x) = x / 19968
normalized_insertions = [f(i) for i in insertions]

insertions_pairs = [(i, beta_dist.distribution_function(i)) for i in normalized_insertions]
plot_b = beta_dist.plot()


No idea how to go about the lower ones. The maximul should be reached at 100, perhaps the parameters for the beta distribution need a little more twiddling?

Currently, it looks like this:


If possible, use sage as a reference for your explanations.

  • $\begingroup$ Doesn't stats.stackexchange.com/questions/1142/… answer your question? $\endgroup$ – whuber Sep 14 '12 at 1:51
  • 2
    $\begingroup$ What are sparks? Are they what I would call spikes? Spikes are values orders of magnitude higher than other points in the time series and they are isolated. If you have spikes do you know why they occur. Are you sure that shrinking them is appropriate? $\endgroup$ – Michael R. Chernick Sep 14 '12 at 3:27
  • $\begingroup$ Yes, I am sure. The spikes represent behavior which is 100% irrelevant to the model. The "real data" has its high points at 100, and between 100 and 300 I want to smooth out the values gradually, exactly like the beta distribution. Whatever is higher than 300 is really, REALLY undesired. These numbers represent "marks" for human behavior and they should represent how well they've done. They are explicitly forbidden to "do more than 300", and as a consequence the mathematical model should reflect that. I've corrected it to spikes, thanks. $\endgroup$ – Flavius Sep 14 '12 at 14:08