Say you have multiple data samples from different days (or some other unit of time) and you want to answer the question: is the distribution for a given day an outlier (compared to other days)? Is there a statistical test that can answer this question?

As a concrete example, say you have the following:

A server that receives requests over time for one of N pages. For a given day, you have a distribution over those N pages (we can assume it's something like a multinomial distribution if that simplifies the problem).

Most days follow the same underlying distribution, but some days are outliers, e.g. a sale day will have a higher percentage of visits to a purchase page.

How can you test a day's empirical distribution against the distribution of other days to determine whether that day's data is an outlier?

In our case there are 1440 data points per day (one per minute).

We would probably compare the candidate day against ~6 other days, but in theory we could go as far back as needed and pick as many days as needed to ensure that the candidate day is not an outlier.

We are choosing a day from which to sample logs for replay load testing, but we want to confirm that the day we're choosing is not an unusual day, e.g. a previous day that we conducted load testing, a sales day, etc. Because we're only sampling logs from the day at a specified rate, the only thing that matters is that the distribution for that day is accurate.

I was thinking something like getting the JS divergence of the day that is being tested with each other day, and then averaging by the number of days that you compared against and seeing if the result is over a threshold, but the problem is that there may be other outlier days in the set of days you test against, so I'm wondering if there's a more principled approach.

  • 1
    $\begingroup$ How many data points per day? How many days total? Are you specifically interested in comparing distributions for the whole day? $\endgroup$ – user2974951 Jan 22 at 8:15
  • $\begingroup$ I updated the post with more information, but the short answers are 1440, ~7 days total, yes. $\endgroup$ – stackedAE Jan 23 at 1:55
  • $\begingroup$ This is going to be tough, you have big samples, so distribution tests will likely reject the null hypothesis of equivalence. You could try a chi square test to compare many distributions. Is there something else that you could test? A mean / median / or something else, which will be more robust? $\endgroup$ – user2974951 Jan 23 at 14:55
  • $\begingroup$ In my case the distributions are multinomial (many requests per day, and each request is randomly one of N different pages according to a the distribution of the day). Do you mean compare the mean of each page? A human can look at each day and determine when a specific day was an outlier/unusual day with a fair degree of confidence, so I imagine there must be some statistical algorithm to determine the same thing. $\endgroup$ – stackedAE Jan 25 at 2:17
  • $\begingroup$ As I've already mentioned, there exist tests to compare distribution, Shapiro-Wilks, Kolmogorov-Smirnov, Chi-square and so on. But given your sample sizes (1440) these tests may not work well, since it is a known fact that they are highly susceptible to noise for big sample sizes. So it may be better to use a different kind of test, for ex. testing the median with a Mann Whitney test, if this makes sense. $\endgroup$ – user2974951 Jan 25 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.