Alternative to Chi-squared test to check if categorical distribution in two sets are the same

Question

I have expected frequencies in each category as shown below:

So my initial data: category - and number of observation of it.

There are too many observations, this why:

As the sample size grows the null hypothesis will be rejected and the p value goes to zero for any small but nonzero deviation from the null hypothesis. With counts, i.e. total number of observations, of more than 50,000 the proper hypothesis test will most likely reject even small differences that are statistically significant but irrelevant in applications.

Anyway I have applied a Chi-squared test to this K-2 contingency, and as a result get a p_value = 6.3723954051318158e-126 . This is not so bad considering that applying this test to completely irrelevant data sets - the p value will be zero.

Using F-test as I know I will get the same result.

Another approach that comes to mind is applying a binomial test. I did it this way:

 stats.binom_test(1500, store.answered.sum(), 0.0233, alternative = 'two-sided')  
# 1500 observed amount, store.answered.sum() - sum over all observations, 0.0233 - expected frequency of that label 

P_value = 0.00023472778370252812. The result is better, because we don't want to reject null hypothesis. However here is another point, that we have to keep in mind:

One of the underlying assumptions is that all observations are drawn independently from the same distribution. This will not hold if there is correlation within a store or heterogeneity in the probabilities/distribution. In those cases the variance assumption of the multinomial/binomial/Poisson model would not hold and we get either under or over dispersion

Generally I can make such an assumption, but I'm not sure.

So my question is, how can we check that data distribution within these data sets are the same? My final goal is to check that the second smaller data set is not shifted (English term may be different) to the bigger original one.

Answer 1

Tests for equivalence test the null hypothesis that quantities are different by a threshold of relevance—the smallest value that researchers, or regulators in the case of, for example, the FDA, consider to be meaningful—and rejection of this null hypothesis is to conclude that the quantities are equivalent within the bounds of the relevance threshold.¹

One form of equivalence tests is the two one-sided tests (TOST) approach, where (typically) two one-sided t or z tests are constructed around the relevance threshold in the upper and lower directions… rejecting both one-sided tests implies that the true value ought to be inferred to lying within the equivalence range. However, TOST, why relatively straightforward to compute, and widely used, ignores an accurate accounting of power to reject by ignoring the non-centrality parameters that come into play in their test statistics. By contrast uniformly most powerful (UMP) tests for equivalence account for such, and provide optimal statistical power to reject equivalence null hypotheses.

Chapter 9, section 9.2 of Welleck's Testing Statistical Hypotheses Of Equivalence And Noninferiority, Second Edition provides a uniformly most powerful test for equivalence for contingency table $\chi^{2}$ tests (or, test for 'collapsability' as the contingency table equivalence testing literature has it). The math for constructing the UMP contingency table test statistic is a tad hairy (by which I mean I haven't learned it yet :), but Welleck includes an R macro for the test and an example application.

Finally, I will note that only testing for difference, or only testing for equivalence implies—without explicit a priori power analysis and justification of minimum relevant effect size—committing to confirmation bias by privileging the direction of evidence/burden of proof. A savvy way to counter that commitment in a frequentist analytic context is to conduct both tests for relevance and tests for equivalence, and draw conclusions accordingly (see the [tost] tag info page for more details on this point).

¹ Relevance thresholds can be asymmetric: closer to 'no difference' in one direction than the other.

Answer 2

I think identifying an appropriate threshold that indicates a meaningful rather than simply a statistically significant difference between your two samples would be a valuable step as described, in part, by @Alexis's answer.

I would like to propose an alternative approach of sorts, though, one based on simulation. The logic here is that you can create a range of plausible sample counts based on your larger dataset, and then determine whether your observed counts for the smaller dataset generally fall inside or outside those plausible ranges.

Using your counts from your larger sample to represent something closer to your population counts then, you can generate a sufficiently large number of random samples from said (pseudo)population of the same size as your smaller sample. I will illustrate using R, and with a much smaller set of categorical data:

> #Observed frequencies in the larger sample:
> lambdas<-c(2500,30000,25000,17000,18750,19200, 2000, 2500, 950, 750)
> N<-sum(lambdas)
> #Total "psuedo"-population size
> N
[1] 118650
> 
> #Probabilities for each category (based on "pseudo"-population)
> p<-lambdas/N
> p
 [1] 0.021070375 0.252844501 0.210703751 0.143278550 0.158027813 0.161820480
 [7] 0.016856300 0.021070375 0.008006743 0.006321113
> 
> #Sample size for smaller data set
> N2<-2500
> 
> #Category names 
> cat.names<-paste('cat', sep='_', letters[1:length(p)])
> 
> #Simulate category counts
> n.sims<-10000
> sim.counts<-data.frame()
> for(i in 1:n.sims){
+ temp<-as.vector(table(sample(cat.names, size=N2, prob=p, replace=T)))
+ sim.counts<-rbind(sim.counts, temp)
+ }
> 
> colnames(sim.counts)<-cat.names
> head(sim.counts)
  cat_a cat_b cat_c cat_d cat_e cat_f cat_g cat_h cat_i cat_j
1    46   576   535   348   453   400    50    49    28    15
2    46   603   537   338   421   426    38    50    25    16
3    50   633   495   350   391   450    46    46    22    17
4    60   606   521   344   440   397    50    50    18    14
5    42   630   539   381   386   398    34    58    19    13
6    48   663   514   356   398   380    40    62    22    17
> 
> #create empty vectors to hold upper and lower percentile values
> LB.95<-vector()
> UB.95<-vector()
> #calculate 95% interval
> for(i in 1:length(p)){
+ LB.95[i]<-quantile(sim.counts[,i], .025)
+ UB.95[i]<-quantile(sim.counts[,i], .975)
+ }
> 
> cbind(cat.names, LB.95, UB.95)
      cat.names LB.95 UB.95
 [1,] "cat_a"   "39"  "67" 
 [2,] "cat_b"   "590" "675"
 [3,] "cat_c"   "487" "566"
 [4,] "cat_d"   "324" "392"
 [5,] "cat_e"   "360" "431"
 [6,] "cat_f"   "369" "442"
 [7,] "cat_g"   "30"  "55" 
 [8,] "cat_h"   "39"  "67" 
 [9,] "cat_i"   "12"  "30" 
[10,] "cat_j"   "9"   "24" 

Now the biggest caveat here is that I am treating estimates from my larger sample ($N$ = 118,650) as if they were parameters from the population. In some ways this simulation then is something of a poor man's Bayesian approach to resolving the problem, where I ignore my uncertainty about the the true parameters based on the large initial sample. One could certainly take a more fully Bayesian approach to this problem, and I am sure there are a number of advocates in the applied statistics community who would see this question as extremely well-suited to Bayesian techniques.

The caveat noted, how do you use this analysis? Well, you can take an obtained sample of size N2 (in my case $N_2$ = 2500), calculate your counts for each category, and determine whether those counts fall within a pre-identified interval based on the simulations (I chose a 95% confidence interval - displayed in the final table).

Note that this approach will not have the nice, clean decision-making rules often relied upon within a hypothesis-testing framework, and depending on your final audience, this may be a non-trivial concern. However, you can answer (perhaps more meaningfully even) whether obtained counts from a smaller sample fall within a likely range of values if the parameters for the population from which that sample was drawn are equivalent to a MUCH larger (presumably previously obtained) comparison sample.