Say you have multiple data samples from different days (or some other unit of time) and you want to answer the question: is the distribution for a given day an outlier (compared to other days)? Is there a statistical test that can answer this question?
As a concrete example, say you have the following:
A server that receives requests over time for one of N pages. For a given day, you have a distribution over those N pages (we can assume it's something like a multinomial distribution if that simplifies the problem).
Most days follow the same underlying distribution, but some days are outliers, e.g. a sale day will have a higher percentage of visits to a purchase page.
How can you test a day's empirical distribution against the distribution of other days to determine whether that day's data is an outlier?
In our case there are 1440 data points per day (one per minute).
We would probably compare the candidate day against ~6 other days, but in theory we could go as far back as needed and pick as many days as needed to ensure that the candidate day is not an outlier.
We are choosing a day from which to sample logs for replay load testing, but we want to confirm that the day we're choosing is not an unusual day, e.g. a previous day that we conducted load testing, a sales day, etc. Because we're only sampling logs from the day at a specified rate, the only thing that matters is that the distribution for that day is accurate.
I was thinking something like getting the JS divergence of the day that is being tested with each other day, and then averaging by the number of days that you compared against and seeing if the result is over a threshold, but the problem is that there may be other outlier days in the set of days you test against, so I'm wondering if there's a more principled approach.