Which statistical test and measure of association should I use in the following situation?
I have a rather large dataset (>30 million obs.). I have a binary variable, and a bunch of other variables (most of them ordinal variables, but some nominal or categorical variables as well). I want to use a test to determine whether or not there seems to be an association between the binary variable and each of the other categorical/ordinal variables.
So here are frequency/contingency tables for two of my variables.
x <- structure(c(873418, 269549, 512178, 184685, 757349, 367854, 1148209,
620801, 1711981, 1053272, 2253784, 1624884, 3102083, 2341371,
2485241, 2509454-439868, 2428430, 2276198, 1379132, 1457545, 554876,
712547, 233438, 276019), .Dim = c(2L, 12L), class = "table", .Dimnames = list(
c("FALSE", "TRUE"), c("00", "01", "02", "03", "04", "05",
"06", "07", "08", "09", "10", "11")))
x
y <- structure(c(23242, 16661, 30792, 22719, 55919, 40112, 89509,
65484, 183195, 120054, 329017, 259172, 1542564, 1198699, 8304387,
6687968, 6380851, 4540228, 500643, 303214), class = "table", .Dim = c(2L,
10L), .Dimnames = list(c("FALSE", "TRUE"), c("01", "02", "03",
"04", "05", "06", "07", "08", "09", "10")))
y
When I inspect the variables visually, it seems clear that the first one (x) has a relationship, while such a relationship is not clear at all on the second one (y). Here's the code I am using to plot the fraction of TRUEs, for each category.
ggplot2::qplot(y = x[1,] / apply(x, 2, sum), ylim = c(0, 1))
ggplot2::qplot(y = y[1,] / apply(y, 2, sum), ylim = c(0, 1))
I would like a test that I can apply to all other variables and would help me identify those cases where there is a relationship (such as x) and where there is not (like in y).
So, my first (naive) try was to use chi-square test. But it fails to distinguish what I want. In fact, the test suggest that both variables have an association with the binary variable (p-value is virtually zero in both cases).
> chisq.test(x)
Pearson's Chi-squared test
data: x
X-squared = 639480, df = 11, p-value < 2.2e-16
> chisq.test(y)
Pearson's Chi-squared test
data: y
X-squared = 36036, df = 9, p-value < 2.2e-16
Clearle che-square is not appropriate here, I would say in this case mainly because:
- It "is sensitive to sample size", and given the large dataset I have, basically the test would say that every variable has an association with the binary variable.
- And also, it "does not give us much information about the strength of the relationship or its substantive significance in the population". And that's something that I want, a measure that indicates the strength of the association.
I appreciate any guidance on what are the most appropriate tests and measures of association here.
Sor the statistical tests, I haven't found anything that could work for my case (everywhere I look they suggest chi-square, but due to the weaknesses mentioned above, that does not work in this case).
As for the measures of association, I've read that I could use Lambda, Cramer's V or gamma. But I am not familiar with those measures so I would like to ask for help here as well. But I've been trying to use them and still have questions. For example.
I am using the R
package DescTools
, which has a lot of those measures. Ad for example, here's the result using Cramer's V:
> DescTools::CramerV(x)
[1] 0.1443388
> DescTools::CramerV(y)
[1] 0.03426402
Ok, so it seems about right that the value of the measure for x is higher than for y. But the help files say that "A Cramer's V in the range of [0, 0.3] is considered as weak". But it seems to be the association for x is actually pretty strong (the fraction of TRUEs drops consistently as x increases, reducing the fraction in about 30 percentage points -from 76% to 45%-). So it seems odd to me that the measure suggests there is a weak association. Also, I am not sure how big of a mistake it would be to use Cramer's V, given that it is for nominal variables (so I would be ignoring the fact that I have ordinal variables -categorical variables whose order have a meaning-).
Sorry for such a long post and I thank you in advance for any advise you could give me about which i) statistical tests and ii) measures of associations to use in this case.