Can I use chi.squared() for effectives?

Question

I have a table with male and female columns and their treatment (1, 2, and 3) by fractions or count. Initially, I used a chi-squared test to compare proportions between the two genders, and the results indicated no significant differences:

                  Female   Male
Treatment_1      0.5569786  0.7199256
Treatment_2      0.1382645  0.0551989
Treatment_3      0.3047569  0.2248755

    Pearson's Chi-squared test

data:  contingency_table
X-squared = 0.068507, df = 2, p-value = 0.9663

However, when I used the same test with the actual counts instead of proportions, it showed significant differences:

               Female      Male
Treatment_1     882        6662
Treatment_2     244        504
Treatment_3     615        2422

Pearson's Chi-squared test

data:  contingency_table
X-squared = 305.25, df = 2, p-value < 2.2e-16

This inconsistency has left me uncertain about whether the chi-squared test is appropriate for my data. Could a prop.test() be a better option? If so, how should I use it?

Additionally, I'd like to compare treatments within each sex to see if there are significant differences between treatments proportions. Can I use a chi-squared test for that comparison?

Answer 1

First, dipetkov is right about the two tests you have run. The first one is invalid and doesn't test anything. The second one tests a null hypothesis that males and females were equally likely to get the different treatments. The null is rejected at a very low p value, indicating that, if they had been randomly assigned to treatment, it would be extremely unlikely to get this pattern.

Second, if you want to test effectiveness (as indicated by your title), you will want some kind of regression. Which kind depends on how effectiveness is measured. If it's a continuous measure, then OLS would be a starting place; if it's categorical, then some form of logistic; if it's survival, perhaps a Cox proportional hazards. And so on.

For any of those methods you would want to include sex and treatment as independent variables. You would also want to include their interaction (to ask whether the treatment is differently effective in males and females) and probably some relevant covariates. You don't say what these groups are for or what's being treated, but it's hard to imagine a situation where there would be no relevant covariates.

Also, you will want to investigate why the proportions assigned to different treatments varied so much.

Answer 2

"Could a prop.test() be a better option? If so, how should I use it?"

Well, they (the prop.test and chisq.test functions in R) both give the same results, depending on your hypothesis and how your data are formatted (see this answer for details). For your data:

> Females <- c(882, 244, 615)
> Males <- c(6662, 504, 2422)

> data <- cbind(Females, Males)
> rownames(data) <- paste("Treatment", 1:3)

> X <- chisq.test(data); X
Pearson's Chi-squared test'

data:  data
X-squared = 305.25, df = 2, p-value < 2.2e-16

> prop.test(data)

3-sample test for equality of proportions without continuity correction

data:  data
X-squared = 305.25, df = 2, p-value < 2.2e-16
alternative hypothesis: two.sided
sample estimates:
   prop 1    prop 2    prop 3 
0.1169141 0.3262032 0.2025025 

So it doesn't matter which one you use. Or does it?

You need to be careful how you state you're null hypothesis.

"...to compare proportions between the two genders...".

This can be interpreted in several ways. I'll show you one or two examples using prop.test.

The proportion test in R is achieved using prop.test(x, n, p=NULL, ...), where x is the vector of counts of successes and n is the corresponding vector of counts of trials. This tests whether the given proportions are equal, or equal to a specified proportion, p.

For example, to test whether the proportions of females (among all individuals) allocated to each treatment are equal, you can type the following code in R:

> prop.test(Females, n=Males + Females) #1

    3-sample test for equality of proportions without continuity correction

data:  Females out of Males + Females
X-squared = 305.25, df = 2, p-value < 2.2e-16
alternative hypothesis: two.sided
sample estimates:
   prop 1    prop 2    prop 3 
0.1169141 0.3262032 0.2025025

The hypothesis is clearly rejected and you can do a similar test for the males. Alternatively, you can test whether the proportions of females allocated to each treatment are equal to the corresponding proportions in males.

> prop.test(Females, n=Males + Females, p=Males / (Males + Females)) #2

    3-sample test for given proportions without continuity correction

data:  Females out of Males + Females, null probabilities Males/(Males + Females)
X-squared = 49961, df = 3, p-value < 2.2e-16
alternative hypothesis: two.sided
null values:
   prop 1    prop 2    prop 3 
0.8830859 0.6737968 0.7974975 
sample estimates:
   prop 1    prop 2    prop 3 
0.1169141 0.3262032 0.2025025

Again, this is rejected. However, this test is definitely not appropriate for your experiment because the number of males and females was not equal.

A seemingly more appropriate test would be one that tests whether the proportions of treatments allocated to females (or males) separately were all equal. We could try:

> prop.test(Females, rep(sum(Females), 3)) #3

    3-sample test for equality of proportions without continuity correction

data:  Females out of rep(sum(Females), 3)
X-squared = 530.71, df = 2, p-value < 2.2e-16
alternative hypothesis: two.sided
sample estimates:
   prop 1    prop 2    prop 3 
0.5066054 0.1401493 0.3532453

This test is rejected. However, I question the validity of this test because the assumptions are violated (the 3 samples are not independent).

As an aside, I note that your proportions do not match those given in the output above. hmmm... Anyway, another hypothesis that may be closer to the one that you want is: "Do the proportions of treatment allocations for females equal the corresponding proportions for males (or vice versa)"?

> prop.test(Females, rep(sum(Females), 3), p=Males / sum(Males)) #4

    3-sample test for given proportions without continuity correction

data:  Females out of rep(sum(Females), 3), null probabilities Males/sum(Males)
X-squared = 652.44, df = 3, p-value < 2.2e-16
alternative hypothesis: two.sided
null values:
    prop 1     prop 2     prop 3 
0.69482687 0.05256571 0.25260743 
sample estimates:
   prop 1    prop 2    prop 3 
0.5066054 0.1401493 0.3532453

The test is rejected. A similar test can be done for males (do the proportions of treatments for males differ compared to females). You'll get a similarly small p-value, but not the same chi-squared statistic. I'll show the output just for completeness.

> prop.test(Males, rep(sum(Males), 3), p=Females / sum(Females)) #5

    3-sample test for given proportions without continuity correction

data:  Males out of rep(sum(Males), 3), null probabilities Females/sum(Females)
X-squared = 2394.3, df = 3, p-value < 2.2e-16
alternative hypothesis: two.sided
null values:
   prop 1    prop 2    prop 3 
0.5066054 0.1401493 0.3532453 
sample estimates:
    prop 1     prop 2     prop 3 
0.69482687 0.05256571 0.25260743 

In these five examples, we are specifically testing given proportions against alternatives. The chi-squared test is a global test of association between two variables (here, sex and treatment). The result (see above) is that there is significant imbalance in this data. A logical next question is: "Where is this imbalance"?

The chi-squared test can also help you answer this question.

> X <- chisq.test(data)

> X$expected
              Females    Males
Treatment 1 1159.3348 6384.665
Treatment 2  114.9500  633.050
Treatment 3  466.7152 2570.285

> X$stdres
               Females      Males
Treatment 1 -15.317735  15.317735
Treatment 2  13.538420 -13.538420
Treatment 3   8.721043  -8.721043

The above output tells you that Treatment 1 had the biggest imbalance for females (too few), followed by Treatment 2 (too many) and then Treatment 3 (too many). Any residual greater than +/- 2 would be considered larger than expected if the null hypothesis was true. You can also visualise the results:

> mosaicplot(t(data), shade = T, main="My experiment")

The plot shows that not enough females were given Treatment 1 and not enough males given Treatment 2.

For your second question:

I'd like to compare treatments within each sex to see if there are significant differences between treatments proportions.

I guess this was answered in #3 above for females. Or perhaps you wanted to test whether the proportions of males and females within each treatment were equal. Be careful of type I errors with repeated testing.

# Treatment 1
    > prop.test(c(882, 6662), n=c(sum(Females), sum(Males)))

    2-sample test for equality of proportions with continuity correction

data:  c(882, 6662) out of c(sum(Females), sum(Males))
X-squared = 233.79, df = 1, p-value < 2.2e-16
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.2137893 -0.1626537
sample estimates:
   prop 1    prop 2 
0.5066054 0.6948269 

Again note that I get different proportions than you got. Other options after obtaining a significant chi-square test result are ransacking and partitioning.