Chi square test between samples with very different size I am to estimate the chi square between two samples of size N and M, where in general N << M (e.g. N=100,000  M=10,000,000). In turn, this affects the uncertainty of each sample. I wonder if there is a way to take this difference into account to quantify the agreement so that one can show that a good chi2/dof with N small can correspond to a large chi2/dof when N ~ M just by e.g. a statistical fluctuation.
Cheers,
Riccardo 
 A: Suppose you are comparing two dice to see if they have the same probabilities
of showing faces 1 through 6. 
Suppose Die A is biased so that face 6 is more likely than face 1, specifically the probabilities of the six faces are given by $p_a = (1,2,2,2,2,3)/12.$
And Die B is a fair die.
If you roll Die A $m = 10,000,000$ times and you roll Die B $n = 100,000$ times, then you might get a table such as MAT below, where rows are
counts for the two dice and columns are faces 1 through 6. We use R to simulate the counts.
p.a = c(1,2,2,2,2,3)/12;  a = sample(1:6, 10^7, repl=T, prob=p.a)
b = sample(1:6, 10^5, repl=T);  # fair die
tab.a = tabulate(a)
tab.b = tabulate(b)
MAT = rbind(tab.a, tab.b);  MAT
        [,1]    [,2]    [,3]    [,4]    [,5]    [,6]
tab.a 834537 1668374 1667463 1666285 1664984 2498357
tab.b  16884   16479   16819   16791   16297   16730

Not surprisingly, a chi-squared test detects that the two dice are
not 'homogeneous' as to face probabilities. The P-value is very nearly $0,$
so the null hypothesis is overwhelmingly rejected at the 5% level.
When the data argument of the R procedure chisq.test is a matrix, the procedure performs a test of homogeneity on the counts in the matrix.
chisq.test(MAT)

        Pearson's Chi-squared test

data:  MAT
X-squared = 11284, df = 5, p-value < 2.2e-16

However, if we know the biased probabilities of Die A, the test above is almost
the same as a goodness of fit test for the data from Die B to the
probabilities in $p_a.$
When the data argument of the R procedure chisq.test is a vector of counts and a separate vector of probabilities is given, the procedure performs a test of goodness-of-fit of the data vector to the probability vector.
chisq.test(tab.b, p=p.a)

        Chi-squared test for given probabilities

data:  tab.b
X-squared = 11522, df = 5, p-value < 2.2e-16

The two chi-squared statistics, while arising from different formulas,
are very nearly the same, and the degrees of freedom are the same, so
the P-value is very nearly the same.
More broadly, in many kinds of two-sample tests where the sample sizes are grossly
unequal, it is as if the larger sample becomes (almost) a fixed standard,
and the smaller sample is judged against that standard. 
For example, if we are trying to find the smaller sample size necessary to achieve a given power, when the larger sample is huge, then the power depends almost entirely on the smaller sample size.
A: You are correct that even if both of your sample sets are drawn from the exact same population, the larger sample size will have a more significant p-value for whatever your comparison happens to be. What you can compare, however, is the effect size, which you can measure using an odds ratio (OR). The OR between your large and small samples should be the same within sampling error, although your larger sample will have a smaller confidence around the OR estimate and hence the smaller p-value. You could do a hypothesis test comparing the OR from your large and small sample sets, and if there's no significant difference between them, then you've shown that the effect size is equivalent in both, even though you have a smaller p-value for the larger sample set.
