I work in an online security setting. My goal is to detect if the number of locked accounts per time unit is stable or not. I've tried several approaches, detailed below, but I am not satisfied yet. One of the challenges is to deal with fluctuating amounts of traffic.

If total traffic would be constant over time, then a Poisson distribution could help out. The locked accounts can be seen as arrivals. Since traffic fluctuates heavily over a day, it is difficult to test per minute or hour. Testing per day could be an option. Holiday periods and weekends could show too low observed numbers, but I am mostly interested in increases.

A change detection algorithm like ADWIN could also help out. If applied on day totals, the assumption would be that the per day the number locked accounts should be stable. Days with lower traffic could be a problem like weekends or holidays; or rather the day after the weekend that show an increase. A proportion seems more to the point.

To keep track of the sample size, a chi-square test could help out. One could test proportions independent of sample size as long as cells are filled with more than 5 observations. The observations of the last hour could easily be compared to the observations of the last week since the chi-square test takes the proportions into account. Some bonferroni stuff should be in place.

I've tested a neural network for baseline traffic prediction, but this method seems unstable. It basically uses a predictor, the error variance and an error outlier rule. This should be state of art. The reason why it not so stable is that the traffic varies wildly per minute, increases and decreases of 4 times seems normal. Could averaging over hours instead of minutes make this model work better?

Would anyone have further suggestions?

  • $\begingroup$ Would you mind posting a small part of data, say data per minute for 23 hours? $\endgroup$ – kjetil b halvorsen Apr 30 '15 at 12:33
  • 2
    $\begingroup$ Sure, see: pastebin.com/yguE6J7K The second column is the frequency (now of total request, not locks), the last column the minute. The first column is an id 1,2,3, the other columns refer to day of week or hours. Notice a lot of 0ś at night. I get the fealing that size of the fluctuation compared to the size of the frequencies is to high. $\endgroup$ – spdrnl Apr 30 '15 at 12:51
  • 2
    $\begingroup$ I think statistical quality control tools, which are aimed at monitoring a manufacturing process and checking whether the process output is within the specified specification or stochastic bounds, may be of help, so I added the tag. $\endgroup$ – StasK Apr 30 '15 at 14:42

Having a look at the data, only the freq variable (second column). Since it is a count, analyzing its square root. The autocorrelation function decays almost linearly to about zero at about lag 400, but that must be an artefact of all the nightly zeros. So, analysing only the block of data up to the first block of zeros, freq[1:934], the autocorrelation function is more sensible, nonzero only at the first lag, with value there about 0.3. Plotting that part of the series, it is strange, almost constant with values about 20, with a few values much larger than that, coming in bursts. So describing the series with an autocorrelation function is not a very good description, maybe some kind of hidden Markov process would give a better model, with one almost constant regime, and another bursting regime, and then the zero-regime. You could try that!


Answering question in comments: count data are often poisson-distributed, at least, that is the simplest model for counts. And, the square root is the variance-stabilizing transformation for the Poisson distribution family. That is, while we have that, for $X\sim \text{Po}(\lambda)$, that $\text{Var}(X)=\lambda$, we have that $\text{Var}(\sqrt{X})\approx \frac14$, approximately independent of $\lambda$. And, many statistical methods, like least squares and most time-series models like ARIMA, works best for data which have a variance independent of its mean.

  • 1
    $\begingroup$ Kjetil, thanks for the autocorrelation verification. Your observations support my own; that there are actually three regimes going on, of which the bursts are the most unreliable. This is also why the neural net approach did not work out (Taken from Ellis' Real Time analytics). Analysing the square root of counts is new to me, could you elaborate on that? $\endgroup$ – spdrnl May 6 '15 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.