What kind of test to use for 2 discrete variables?

Question

I'm trying to carry out a hypothesis test for data that has the following structure.

Total number of correct answers out of 6 questions for each subject that only has a HS diploma:

3, 2, 3, 5, ... (there are 16 total subjects)

Total number of correct answers out of 6 questions for each subject that has a College degree:

5, 5, 3, 6, ... (there are 12 total subjects)

I'm not sure if this is relevant for my question but the data was obtained by asking 28 random subjects their education level and then recording how many out of 6 questions they answered correctly.

I used a 2 sample t-test to see if the population means differ for both groups. Is there a better method for this type of data? I'm asking because the data is discrete and I'm not sure if the t-test is best suited for this kind of data.

Also the t-test would tell me if the population means differ but lets say the question of interest is whether or not education (HS vs College) is associated with literacy (total number of correct questions), what kind of test would best answer that question?

Thank you.

Answer 1

Right, considering the modest sample sizes and discreteness of the data, there could be some question about applying the two-sample t test here. The t test formally assumes that the data is normally distributed, which of course is not true here since the data is discrete. For large sample sizes, the t test is robust against violations of the normality assumption, but for smaller sample sizes it is less so. It may still be all right here, since the sample size is not excessively small, and especially because the data is bounded (between 0 and 6) so there can be no concerns about heavy tails or outliers (which otherwise would pose the biggest threats to the accuracy of the t test). The Wilcoxon rank sum test is an alternative which could certainly be applied, and which would avoid any concerns about distributional assumptions, although it may provide slightly less power than the t test. If you use the Wilcoxon rank sum test here, be sure to use an implementation that properly handles ties, as there are many ties here.

On the other hand, summarizing the data as counts of correct answers already discards some potentially valuable information. Particularly if the questions have varying degrees of difficulty and quality, you may be able to get more out of the data and construct a more powerful hypothesis test by applying a method based on item response theory, although this would require a more complex analysis.

Edit: As another answer mentions, the chi-square test is another option here. A drawback of the chi-square test is that it ignores the ordering of the 7 possible responses (0-6), which could lead to a less powerful test. The results of the chi-square test are also less straightforward to interpret, since if the test rejects the null hypothesis, it might be because the two populations have different means, or it could be because they have different variances, or because there is some other kind of difference between the two distributions. Another way of looking at it is that the chi-square test has a more specific null hypothesis, namely that the two distributions are identical, compared to the null hypothesis of the t test that the two distributions have the same mean. The consequence of this is that the chi-square test also has a broader alternative hypothesis, one which is less focused on detecting differences in the means, and this explains why it would be expected to be less powerful if differences in the means is what you are looking for.

Answer 2

Let's take a look at some simulations :) For example, I've checked a binary variable applying t-test, bootstrap, chi-square test and z-test for proportions.

I've measured power by generating multiple random samples of 1 and 0 as:

data_1 = np.random.choice([0, 1], p=[1-p1, p1], size=100)
data_2 = np.random.choice([0, 1], p=[1-p2, p2], size=100)

So, p1 and p2 define shares of 1-s in two samples.

Bootstrap is made like:

def boot_mean(x):
    return np.random.binomial(p=x.mean(), n=x.shape[0], size=10000) / x.shape[0]

T-test and chi-square test have been taken from scipy.stats. Z-test has been implemented by myself.

So, I have generated a lot of samples (10 000 iterations, generating two independent samples on each) and checked in what share of iterations each test shows p-value <= 0.05.

For p1 = 0.1, p2=0.12 it seems like t-test and z-test perform alike and show much less power than other tests, but work well for big enough samples.

There is something strange about chi-square test as it shows much more false positives for p1 = 0.5, p2 = 0.5 (two fair coins) case:

And the strange thing is that during the simulation series of 10000 runs for sample size = 1000 chi-square test shows a strange p-value distribution:

while it should be a near-uniform one.

Just to compare, other tests show much more reasonable p-value distributions:

Link to a notebook (sorry, it's little messy).

Answer 3

This seems like a typical situation for the Chi-squared test for independence. It tests whether two categorical properties are dependent or not.

In your case Property $A=\{HS, College\}$ and Property $B = \{0,\dots,6\}$.

Your null hypothesis $H_0$ would be $P(a,b) = P_A(a)P_B(b)$ (so not dependent) and your alternative $H_1$ would be $P(a,b) \neq P_A(a)P_B(b)$ (so dependent).

The idea of the test is the following. It compares the frequencies of your sample regarding property A and B with the expected ones where the two properties would be independent.

For example, you would have 5 people from HS with 4 right answers. And in total 7 people who had 4 answers right. So you would compare $$\frac{\text{amount of people with }A=HS, B=4}{\text{total amount of people}} = \frac{5}{28} \approx 0.18$$ with $$\frac{\text{amount of people with } A = HS}{\text{total amount of people}} \times \frac{\text{amount of people with } B=4}{\text{total amount of people}} = \frac{16 \times 7}{28\times28} \approx 0.15$$

It turns out there is a statistic using this frequencies which converges towards the $X^2_{(a-1)(b-1)}$ distribution (a Chi square with $(a-1)(b-1)$ degrees of freedom, where $a = |A|$ and $b = |B|$). So $a = 2$ and $b = 7$ for your scenario.

Check out this source: https://stattrek.com/chi-square-test/independence.aspx for the formulas and more explanations.