association measure and statistical test for binary and ordinal variables in large datasets

Question

Which statistical test and measure of association should I use in the following situation?

I have a rather large dataset (>30 million obs.). I have a binary variable, and a bunch of other variables (most of them ordinal variables, but some nominal or categorical variables as well). I want to use a test to determine whether or not there seems to be an association between the binary variable and each of the other categorical/ordinal variables.

So here are frequency/contingency tables for two of my variables.

x <- structure(c(873418, 269549, 512178, 184685, 757349, 367854, 1148209, 
620801, 1711981, 1053272, 2253784, 1624884, 3102083, 2341371, 
2485241, 2509454-439868, 2428430, 2276198, 1379132, 1457545, 554876, 
712547, 233438, 276019), .Dim = c(2L, 12L), class = "table", .Dimnames = list(
    c("FALSE", "TRUE"), c("00", "01", "02", "03", "04", "05", 
    "06", "07", "08", "09", "10", "11")))
x

y <- structure(c(23242, 16661, 30792, 22719, 55919, 40112, 89509, 
65484, 183195, 120054, 329017, 259172, 1542564, 1198699, 8304387, 
6687968, 6380851, 4540228, 500643, 303214), class = "table", .Dim = c(2L, 
10L), .Dimnames = list(c("FALSE", "TRUE"), c("01", "02", "03", 
"04", "05", "06", "07", "08", "09", "10")))
y

When I inspect the variables visually, it seems clear that the first one (x) has a relationship, while such a relationship is not clear at all on the second one (y). Here's the code I am using to plot the fraction of TRUEs, for each category.

ggplot2::qplot(y = x[1,] / apply(x, 2, sum), ylim = c(0, 1))
ggplot2::qplot(y = y[1,] / apply(y, 2, sum), ylim = c(0, 1))

I would like a test that I can apply to all other variables and would help me identify those cases where there is a relationship (such as x) and where there is not (like in y).

So, my first (naive) try was to use chi-square test. But it fails to distinguish what I want. In fact, the test suggest that both variables have an association with the binary variable (p-value is virtually zero in both cases).

> chisq.test(x)

Pearson's Chi-squared test

data:  x
X-squared = 639480, df = 11, p-value < 2.2e-16

> chisq.test(y)

Pearson's Chi-squared test

data:  y
X-squared = 36036, df = 9, p-value < 2.2e-16

Clearle che-square is not appropriate here, I would say in this case mainly because:

• It "is sensitive to sample size", and given the large dataset I have, basically the test would say that every variable has an association with the binary variable.
• And also, it "does not give us much information about the strength of the relationship or its substantive significance in the population". And that's something that I want, a measure that indicates the strength of the association.

I appreciate any guidance on what are the most appropriate tests and measures of association here.

Sor the statistical tests, I haven't found anything that could work for my case (everywhere I look they suggest chi-square, but due to the weaknesses mentioned above, that does not work in this case).

As for the measures of association, I've read that I could use Lambda, Cramer's V or gamma. But I am not familiar with those measures so I would like to ask for help here as well. But I've been trying to use them and still have questions. For example.

I am using the R package DescTools, which has a lot of those measures. Ad for example, here's the result using Cramer's V:

> DescTools::CramerV(x)
[1] 0.1443388
> DescTools::CramerV(y)
[1] 0.03426402

Ok, so it seems about right that the value of the measure for x is higher than for y. But the help files say that "A Cramer's V in the range of [0, 0.3] is considered as weak". But it seems to be the association for x is actually pretty strong (the fraction of TRUEs drops consistently as x increases, reducing the fraction in about 30 percentage points -from 76% to 45%-). So it seems odd to me that the measure suggests there is a weak association. Also, I am not sure how big of a mistake it would be to use Cramer's V, given that it is for nominal variables (so I would be ignoring the fact that I have ordinal variables -categorical variables whose order have a meaning-).

Sorry for such a long post and I thank you in advance for any advise you could give me about which i) statistical tests and ii) measures of associations to use in this case.

Answer 1

You are observing a well-known effect of a large sample size on the statistical significance of even small effects. You might for example see this answer:

How to correct for small p-value due to very large sample size

In a nut shell, paraphrasing some of the relevant points, your chi-squared test is not wrong even if it appears so. What you seem to object to is a small effect size being accompanied by high significance. With a large sample size, even small effects can be identified very reliably hence high significance and low p-value. It is not wrong but it is probably not the right question for you to ask.

What you are probably interested in is the effect size compared to how much you believe in the linear model (you assume a linear relationship between the two variables when you run your chi-squared test). If you believe that the linear model is perfectly correct then you must believe your low p-values. But more likely there is systematic error. You might compare the variation in your outcome variable as predicted by the linear regression that corresponds to you chi-square test, as the independent variable takes extreme values in your sample, to the effect size that you would consider minimally meaningful.