Detect dramatic increase in proportion among multiple classes

Question

I would like to use some tool to detect spike in one class's proportion.

Assume I received roughly the same percentage of red, blue and yellow candies throughout time. That means the absolute number can go up and down for each type of candy but the percentage for each candy will stay roughly the same. For example, it could be

1. 80 red (28%), 90 blue (32%), 110 yellow (39%)
2. 1 red (33%), 1 blue (33%), 1 yellow (33%)

And these are totally fine.

I want a tool to statistically detect when a particular percentage is larger than usual. For example, let's say one day i received 5 red (71%), 1 yellow and 1 blue. That will trigger the warning that the percentage for red is abnormal.

My gut instinct told me i should use Chi-square test to test one class vs. the rest of the classes. Is that correct?

Answer 1

First transform your observations into proportions, as you have mentioned. Then you can apply some outlier detection method. Using z-score or Median Absolute Deviation on each class will probably work quite well for a few classes.

Note: for N classes, the proportions can be fully described by N-1 numbers. Since the proportions sum to 100%. So one column is a redundant column and can be dropped.

Answer 2

Your goal is: "I want a tool to statistically detect when a particular percentage is larger than usual." That is what outlier/anomaly detection methods are made for.

Those methods have different techniques for achieving this. There are good libraries available, see e.g. here for python libs. Among the most popular methods, that also work for a large number of classes, are e.g. k-nearest neighbor methods or isolation forest. Note, however, that they rather detect any kind of isolation, which might also be an isolated very small value as for instance the value $0.1\%$ in $(0.1\%, 24.9\%, 25\%, 25\%, 25\%)$. Or it could be an isolated value in the middle. Those anomalies would not give you high values of z-score.

Most anomaly detection methods provide you with an outlier score which is a measure of its "outlier-ness". That makes those methods particularly flexible since you can yourself decide about where to set the threshold, i.e. above which value of the outlier score a value is to be considered an outlier.

Answer 3

So here (below) is a customizable tool (in R) that you can explore your problem with, and it is how I would think about the problem using a bottom-up approach. You can re-run it a few times with the same settings to get a feel for what run-to-run differences look like and what your uncertainty in the estimate of the means might look like.

This link is to a paper on abrupt change detection and is how I like to think about detecting abrupt changes.

Code:

library(tidyverse)
library(matrixStats)

#parameters 
num_boots <- 3000
num_draws <- 10

#make original population 
pop <- sample(x = c("a", "b", "c"), #put your values here
              size=1000,            #total population
              prob=c(1,1,1)/3,      #sum to 1, frequencies of values
              replace = T)

#convert to factor (makes compute convenient)
pop <- as.factor(pop)

#get stats on it
summary(pop)

#prep for loop
log_store <- matrix(NA, num_boots, length(levels(pop))) %>% as.data.frame()

#bootstrapping
for(i in 1:num_boots){
  y <- sample(pop,num_draws)
  
  for(j in 1:length(levels(pop))){
  log_store[i,j] <- length(which(y==levels(pop)[j] ) ) 

  }
}

#make summaries in a useful form
df <- data.frame(labels = levels(pop),
                 sample_mean    = colMeans(log_store),
                 sample_std_dev = colSds(log_store %>% as.matrix()),
                 sample_max = colMaxs(log_store %>% as.matrix()),
                 sample_min = colMins(log_store %>% as.matrix()) )

#print it
print(tibble(df))

My output looks like this:

So what do I get from this?

• mean number per draw should have about the same proportions as in population
• one standard deviation is about 1.5 units so, if I use rules of thumb, about 67% of the time my values should be within 1.5 units of the mean, and about 99% of the my values should be within about 4.5 units of the mean.

To get the 80/90/110 you would change the code to

pop <- sample(x = c("red", "blue", "yellow"), #put your values here
              size=280,            #total population
              prob=c(80,90,110)/280,      #sum to 1, frequencies of values
              replace = T)

If you only draw 7 per day then adjust 'num_draws' to be 7 instead of ten.

The result from that would be this:

From this we see that most of the time we get 2-3 of each. We would find it rather unlikely to get more than about 6 red candies.