I have a stream of real continuous numbers (Figure. 1). The stream flows from left to right. At $t_1$ the first number entered the stream. Every time a new number enters the stream I check if there is a sudden change or high variation.

How can I compute the sudden change or high variation without using any threshold. The following picture is just and example.

enter image description here

What I was doing is simply compute a quantity $Q_x = |t_4 - t_5|$ and if $Q_x > \delta$ then there is a sudden change and a high variation. Using a threshold is wrong in my situation because there is no bound for the real numbers in the stream. Another idea came to my mind which is the following:

$if\;|t_5-t_4| >|t_4-t_3| + \delta $ where $\delta = [0,1]$ then there is a sudden change.

What do you think? Thank you.


when you "sudden" you imply that there's a notion of what is not sudden, i.e. you seem to have some idea of what is a big change. on the other hand you don't want to use threshold, which tells me that you don't want to rely entirely on your preconceived notion of what is a big change. so it seems that you want to combine your current understanding of the big change with incoming data. you could do this in Bayesian style. for instance you come up a prior for a variance, then while the data is incoming you keep updating the variance of the series. then you detect sudden change by evaluating the deviation of each jump against the variance. maybe you set 1% confidence.

  • $\begingroup$ That's exactly it. I am trying to model this problem, but I couldn't come up with a solution. $\endgroup$ – Hani Gotc Mar 5 '14 at 16:08
  • $\begingroup$ @Aksakal, for someone how has little background in statistics, could you please explain your answer? $\endgroup$ – Hassan Sep 29 '15 at 11:24
  • $\begingroup$ @haxan7, let's say you have the simplest model: you think that the process tends to generate numbers around the constant $c$: $x_t=c+\varepsilon_t$, where $\varepsilon_t$ is the random error. Let's say in the next period you got $x_{t+1}$. You compute the deviation $e_{t+1}=x_{t+1}-c$. If you knew the variance of the error $\sigma_\varepsilon^2$, you could compare the deviation $e_{t+1}$ against the variance. For instance, assuming the errors are normal, there's less than 5% chance to have $|e_{t+1}|>1.96\sigma_\varepsilon$, so you could call this a sudden change. $\endgroup$ – Aksakal Sep 29 '15 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.