Apparent paradox in student T-test

Question

Let's say I have 2 series of data:

• series A has 1M samples, out of which 1000 are "1", the rest is 0
• series B has 100 samples, all 0s

I'm trying to use a statistical test to check if boths series have same mean or not. To me, these series could have the same mean (ie 0.001).

Using scipy:

from scipy import stats
A = [0.0]*100
B = [1.0]*1000 + [0.0]*(1000000-1000)
scipy.stats.ttest_ind(A,B, equal_var=False)

Returns a pvalue=1.399064173683964e-219, which means that both series are different. It seems that the size of A is not taken into account, ie I get the same results if I switch A to be 1 million zeroes.

Is this expected? To me, this p-value is inacurate. Have I made an incorrect assumption somewhere?

Answer 1

Student's t-test is based on the normality of the data. You have manually generated binary data and I think it cannot be assumed they are normal.

You should test the probability of being 0 or 1, instead:

$$H_0: p_A = p_B, \text{ } H_A:p_a \neq p_B$$

You can estimate the probability of being 1 in the sample A and B as

$$\hat{p}_A = \frac{\text{number of ones}}{\text{sample size}} = 0$$

$$\hat{p}_B = \frac{\text{number of ones}}{\text{sample size}} = 0.001$$

Using the Moivre-Laplace central limit theorem you get

$$\frac{\hat{p}_A - \hat{p}_B}{\sqrt{\frac{\hat{p}_A (1 - \hat{p}_A)}{n_A} + \frac{\hat{p}_B (1 - \hat{p}_B)}{n_B}}}$$

is approximately $N(0,1)$ standard normal ($n_A$ and $n_B$ are the corresponding sample sizes).

So calculating the above test statistic, if it is greater than $1.96$ or less than $-1.96$, you should reject $H_0$ at significance level $0.05$ (the calculation left as an exercise to the reader :-) ).

Answer 2

Assuming $$H_0: \mu_1 = \mu_2, H_A: \mu_1 \neq \mu_2 $$ and $\alpha = 0.05$.

The t-statistics is calculated by: $$t_{obs} = \frac{\hat{\mu_1}- \hat{\mu_2}}{\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}}$$

In your example, let's calculate the variances: $$\sigma^2_1 = p(1-p)=0.001(0.999) = 0.000999$$ $$\sigma^2_2 = 0$$

Therefore, the t-statistic would be: $$t_{obs} = \frac{0.001- 0}{\sqrt{\frac{0.000999}{1000000}+\frac{0}{100}}} \approx 31.64$$

(Notice that, because you know the variance, this is actually a z-test because $s^2 = \sigma^2$).

Because you have a two-sided test, your critical region is: $$CR = [t < -1.96; t>1.96]$$ Therefore, you reject the null hypothesis and you can't say that the means are equal.