How to perform a statistical significance test between 2 groups of binary data?

Question

I have a binary column where 0 means fail and 1 means success. This column have been grouped by a second column called events. Here's a data sample before grouping:

isPaid  isEvent
   0.0    event
   0.0    event
   1.0    event
   1.0    unknown
   1.0    unknown

I group the data taking into account only the cases where isPaid==0 (failure), because this is what i'm interested on. After grouping, i get the total count of each isPaid value like this:

df[df['isPaid'] == 0].groupby('isEvent')['isPaid'].count()

And this is what i get:

isEvent
event      308991
unknown    251063

How can i test if the difference between the 2 counts is statistically significant?

I have thought about a paired t-test but since I'm using binary data I'm not sure this is the right way to go. Also, these tests would check the mean, but i just want to know if the difference between the counts is statistically significant.

How could I do this?

Answer 1

There are two approaches to reach this equation, from the mean of repeated Bernoulli trials (each trial is one observation x), or from a Gaussian approximation of a Binomial distribution (the number of successes is our only observation).

Mean of repeated Bernoulli trials The $z$-score of the mean is $$z = \frac{\bar{x} - \mu}{\sqrt{\sigma^2/n}}.$$

The Bernoulli distribution has $\mu = p$ and $\sigma^2 = p(1-p)$. The mean, $\bar{x} = \frac{1}{n} \sum_{i = 1}^n x_i$ is just our estimated probaility $\hat{p}$. With $p=p_0$ we get $$z = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}}.$$

Since your $n$ is large, you can use the Central Limit Theorem stating that $\bar{x}$ approaches a Gaussian distribution, and you can find the corresponding $p$-value for your $z$-score in a table for Gaussian distribution.

Gaussian approximation of Binomial distribution The $z$-score for an observation is defined as $$z = \frac{x-\mu_X}{\sigma_X}$$ where $\mu_X$ is the expected value and $\sigma_X$ is the standard deviation of the random variable $X$.

In our case, $x$ is the sum of the outcomes of the $n$ trials, or the count of "event"s to be even more specific. This is how the binomial distribution is defined, with $\mu_X = np$ and $\sigma_X = \sqrt{np(1-p)}$. If we substitute that into the definition of the $z$-score with $p=p_0$ we get

$$z = \frac{x-np_0}{\sqrt{np_0(1-p_0)}}.$$

Now we divide both the numerator and the denominator by $n$ we get

$$ z = \frac{x/n-p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}},$$

but $x/n$ is just our estimated probability $\hat{p}$; the count of "event"s divided by the number of trials.

Since your $n$ is large, you can approximate the binomial distribution with a Gaussian, and you can directly look up your $z$-score in a $p$-value table for Gaussian distribution.

Answer 2

If I understand you correctly, you are asking if the probability of "event" is different from the probability of "unknown" based on the numbers in your last table. You can think of this as n = 308,991+251,063 = 560,054 trials with outcome 1 for "event" and 0 for "unknown" (or the other way around). If the probability for "event" is equal to the probability of "unknown", they are both 0.5. So this reduces to testing whether the probability of "event" is 0.5.

Since your n is really large, you can safely use a Gaussian approximation and calculate the z-score

$$ z = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}} $$

where $\hat{p}$ is your estimated probability of "event" (308991/n) and $p_0 = 0.5$.

When you do these calculations, you will get a really high z-score, corresponding to a very low p-value, and you can safely conclude that the probability of "event" is statistically significantly different from $p_0 = 0.5$.