# Statistical test to see whether we drew overly-balanced samples

I have an urn with $R$ red and $B$ black balls. $R$ and $B$ are large numbers. I draw some $n$ samples of different sizes without replacement until the urn is empty. The sizes of the samples are not a random variable, consider it a known determined value.

I want to see whether I am drawing samples that are more balanced for red and black balls than expected by chance. Which test should I use? I can think of tabulating the numbers and applying the chi-square or G-test. Usually, it tests whether my samples are overly biased, rather than balanced. So I should take the lower tail as the p-value in this case.

Is there a better or standard test for this that I should conduct instead? For my case, the sizes of each draw are small, say between 2 and 10. On the other hand $R$, $B$, and number of draws are large, in the thousands. I definitely don't want to program for combinatorial counting or p-value estimation by Monte Carlo simulations.

Update: So I tried to use R's chisq.test and vcd::assocstats for this, and there is no way to specify that I am interested in the lower tail. The statistics used for the two tests are $\sum_{i,j} \frac{(O_{i,j}-E_{i,j})^2}{E_{i,j}}$ and $\sum_{i,j} \log \frac{O_{i,j}}{E_{i,j}}$. They should be small when $E$ is very close to $O$.

• Chi-squared test is goodness of fit test. If the test shows large p-value, it means there is no large deviation/"overly biased" observation (to use your terminology), hence "overly balanced". It is one sided and you can't take the opposite direction of the test. Aug 10, 2020 at 15:26

This is an interesting question. As noted, the Chi-squared test is traditionally a one-sided test on the upper tail. However, the Chi-squared distribution is two-sided, and it's possible to instead implement a one-sided test on the lower tail.

Tables with fixed margins

However, is the Chi-squared distribution appropriate in this case?

I have an urn with R red and B black balls. R and B are large numbers. I draw some n samples of different sizes without replacement until the urn is empty. The sizes of the samples are not a random variable, consider it a known determined value.

Under the null hypothesis for an $$r \times c$$ contingency table, multinomial sampling and product multinomial sampling asymptotically leads to a Chi-squared distribution, but what is envisioned here is not multinomial sampling but hypergeometric sampling. As whuber notes below, given the dependence inherent in this sort of contingency table, i.e. the expectation for the $$R$$ and $$B$$ cells for a particular sample are conditional on the previous samples, it is not clear that the Chi-squared distribution obtains under the null in this scenario, and one might wonder about serial correlation (for references, see Pesaran and Timmermann).

However, as far as I understand it, the scenario under consideration is simply a $$r \times c$$ contingency table with fixed margins and is identical to the motivating example offered by Alaouf (1987).

Suppose N individuals belonging to c classes (ethnic groups, for example) are hired for N summer jobs which fall into r categories. Both the number of individuals of each class and the number of jobs of each category are fixed. The hypothesis of independence in this context is the hypothesis that individuals are assigned to different job categories without regard to their class.

Substitute the c classes for $$R$$ and $$B$$ and substitute the r job categories of fixed size for the samples of fixed size.

For such a contingency table with fixed margins, under the null, the Chi-squared distribution does in fact obtain with degrees of freedom equal to $$(c - 1)(r -1)$$. For proofs, see Alaouf (1987) and Cheolyong (2007).

Simulation

While the exact details of the OP's case are not provided, we can provide a simple simulation to demonstrate how the Chi-squared statistics follows a Chi-squared distribution under the null.

n <- 1000
rs <- rep(10, 1000)
cs <- c(5000, 5000)

E <- outer(rs, cs) / (sum(rs))

set.seed(48)
a <- r2dtable(n, rs, cs)

sim <- 1:n

for(i in 1:n) {

x <- as.integer(a[[i]])

sim[i] <- sum((x - E) ^ 2 / E)

}

chi <- rchisq(n, df = 999)

ks.test(sim, chi)


The p-value for the Kolmogorov-Smirnov test is ~0.5.

Other simulations, however, demonstrate that contingency tables with a smaller number of observations may not approximate the Chi-square distribution. Thus, for the small table below, we utilize a simulated p-value.

Implementation

The chisq.test in R is a one-sided test on the upper tail, but a few simple modifications to the source code produces a function to test the lower tail. (The source code is available here.)

chisq.test_lower <- function(x, simulate.p.value, B)  {

nr <- as.integer(nrow(x))
nc <- as.integer(ncol(x))
sr <- rowSums(x)
sc <- colSums(x)
E <- outer(sr, sc) / sum(x)

STATISTIC <- sum(sort((x - E) ^ 2 / E, decreasing = TRUE))

if(simulate.p.value == FALSE)  {

PARAMETER <- (nr - 1L) * (nc - 1L)
PVAL <- pchisq(STATISTIC, PARAMETER, lower.tail = TRUE)

}

else {

PARAMETER <- NA

tmp <- .Call(C_chisq_sim, sr, sc, B, E)
PVAL <- (1 + sum(tmp <= STATISTIC)) / (B + 1)

}

structure(list(statistic = STATISTIC,
parameter = PARAMETER,
p.value = PVAL))

}

environment(chisq.test_lower) <- environment(stats::cor.test)


Now let's say there are one hundred balls in the urn, 50 black and 50 red. We proceed to draw 10 samples of 10 balls each. We tabulate the samples to create a 10 x 2 contingency table. Suppose we ended up with 6 draws that included 5 red and 5 black balls as well as a few draws with 4 and 6 and 3 and 7.

x <- matrix(data = c(4, 6, 6, 4, 7, 3, 3, 7, rep(5, 12)),
ncol = 2, byrow = TRUE)

set.seed(11)
chisq.test_lower(x, simulate.p.value = TRUE, B = 2000)


Simulating the p-value produces a p-value of ~0.06. Given that your urn includes very large numbers, this method should give you sufficient statistical power.

(Editing in response to feedback in the comments below.)

• What is special about this situation is that the expected count at each step is conditional on the results of the preceding steps. It is unclear that the usual chi-squared test applies or, when it does, what the degrees of freedom ought to be or what the expectations ought to be. How does your solution address these issues?
– whuber
Feb 28, 2022 at 16:17
• @whuber, my reading of the question was that the OP wanted to carry out a Chi-squared test for each sample. (Of course after each draw, the expected counts would need to be adjusted.) Feb 28, 2022 at 16:56
• @whuber, you are correct. Thanks for pointing this out. I've edited my answer accordingly. Mar 2, 2022 at 21:14
• It differs in several important ways. One is that the values in the table are serially correlated, with negative correlation coefficients. That's enough to raise significant questions about applying any version of a chi-squared test and raises doubts even about the simulation procedure. Another is that your chi-squared statistic looks like it leads to an inadmissible test: this is a technical term for a procedure that is inferior to another (using a better statistic) no matter what the underlying distribution might be.
– whuber
Mar 3, 2022 at 13:36
• Sorry; I had forgotten the details of this question. The margins are indeed fixed, as you write. However, I believe the simulation, the hypergeometric theory, and the chi-squared approximation all assume every possible configuration of the table is equiprobable. I think this explains why the distribution of the chi-squared statistic under this sampling program is not chi-squared. We might differ here over whether to analyze the experiment as described or else to analyze the table conditional on the results. You are doing the latter, whereas I think the former may be more appropriate.
– whuber
Mar 7, 2022 at 13:39