4
$\begingroup$

Which statistical test and measure of association should I use in the following situation?

I have a rather large dataset (>30 million obs.). I have a binary variable, and a bunch of other variables (most of them ordinal variables, but some nominal or categorical variables as well). I want to use a test to determine whether or not there seems to be an association between the binary variable and each of the other categorical/ordinal variables.

So here are frequency/contingency tables for two of my variables.

x <- structure(c(873418, 269549, 512178, 184685, 757349, 367854, 1148209, 
620801, 1711981, 1053272, 2253784, 1624884, 3102083, 2341371, 
2485241, 2509454-439868, 2428430, 2276198, 1379132, 1457545, 554876, 
712547, 233438, 276019), .Dim = c(2L, 12L), class = "table", .Dimnames = list(
    c("FALSE", "TRUE"), c("00", "01", "02", "03", "04", "05", 
    "06", "07", "08", "09", "10", "11")))
x

enter image description here

y <- structure(c(23242, 16661, 30792, 22719, 55919, 40112, 89509, 
65484, 183195, 120054, 329017, 259172, 1542564, 1198699, 8304387, 
6687968, 6380851, 4540228, 500643, 303214), class = "table", .Dim = c(2L, 
10L), .Dimnames = list(c("FALSE", "TRUE"), c("01", "02", "03", 
"04", "05", "06", "07", "08", "09", "10")))
y

enter image description here

When I inspect the variables visually, it seems clear that the first one (x) has a relationship, while such a relationship is not clear at all on the second one (y). Here's the code I am using to plot the fraction of TRUEs, for each category.

ggplot2::qplot(y = x[1,] / apply(x, 2, sum), ylim = c(0, 1))
ggplot2::qplot(y = y[1,] / apply(y, 2, sum), ylim = c(0, 1))

enter image description here

I would like a test that I can apply to all other variables and would help me identify those cases where there is a relationship (such as x) and where there is not (like in y).

So, my first (naive) try was to use chi-square test. But it fails to distinguish what I want. In fact, the test suggest that both variables have an association with the binary variable (p-value is virtually zero in both cases).

> chisq.test(x)

Pearson's Chi-squared test

data:  x
X-squared = 639480, df = 11, p-value < 2.2e-16

> chisq.test(y)

Pearson's Chi-squared test

data:  y
X-squared = 36036, df = 9, p-value < 2.2e-16

Clearle che-square is not appropriate here, I would say in this case mainly because:

I appreciate any guidance on what are the most appropriate tests and measures of association here.

Sor the statistical tests, I haven't found anything that could work for my case (everywhere I look they suggest chi-square, but due to the weaknesses mentioned above, that does not work in this case).

As for the measures of association, I've read that I could use Lambda, Cramer's V or gamma. But I am not familiar with those measures so I would like to ask for help here as well. But I've been trying to use them and still have questions. For example.

I am using the R package DescTools, which has a lot of those measures. Ad for example, here's the result using Cramer's V:

> DescTools::CramerV(x)
[1] 0.1443388
> DescTools::CramerV(y)
[1] 0.03426402

Ok, so it seems about right that the value of the measure for x is higher than for y. But the help files say that "A Cramer's V in the range of [0, 0.3] is considered as weak". But it seems to be the association for x is actually pretty strong (the fraction of TRUEs drops consistently as x increases, reducing the fraction in about 30 percentage points -from 76% to 45%-). So it seems odd to me that the measure suggests there is a weak association. Also, I am not sure how big of a mistake it would be to use Cramer's V, given that it is for nominal variables (so I would be ignoring the fact that I have ordinal variables -categorical variables whose order have a meaning-).

Sorry for such a long post and I thank you in advance for any advise you could give me about which i) statistical tests and ii) measures of associations to use in this case.

$\endgroup$
  • $\begingroup$ Your plots suggest a simple linear model. In the first case the slope (coefficient) is not zero. In the other case it seems to be zero. I could elaborate more on this later if this will help you. Additional information, like standard deviation per case and distributional assumptions would allow more set of models like ANOVA and Tukey's HSD $\endgroup$ – Drey Nov 14 '16 at 16:03
1
$\begingroup$

You are observing a well-known effect of a large sample size on the statistical significance of even small effects. You might for example see this answer:

How to correct for small p-value due to very large sample size

In a nut shell, paraphrasing some of the relevant points, your chi-squared test is not wrong even if it appears so. What you seem to object to is a small effect size being accompanied by high significance. With a large sample size, even small effects can be identified very reliably hence high significance and low p-value. It is not wrong but it is probably not the right question for you to ask.

What you are probably interested in is the effect size compared to how much you believe in the linear model (you assume a linear relationship between the two variables when you run your chi-squared test). If you believe that the linear model is perfectly correct then you must believe your low p-values. But more likely there is systematic error. You might compare the variation in your outcome variable as predicted by the linear regression that corresponds to you chi-square test, as the independent variable takes extreme values in your sample, to the effect size that you would consider minimally meaningful.

$\endgroup$
  • $\begingroup$ thank, this is helpful. A couple of questions, though. 1) Could u provide some references of literature that discusses the issues of small p-values with very large sample size and how to deal with them? (like the discussion you and the other response summarized, but with some papers we can study further). $\endgroup$ – elikesprogramming Nov 17 '16 at 21:34
  • 1
    $\begingroup$ 2) In my case, I am not much interested in this example data I posted ('cause regardless of the p-value the relationship on the first one is very clear and meaningful and the second one not), but on doing something similar for a large number of variables and to -sort of- automatically identify which variables have a clear association with my response/dependent variable. Any advise for this?, ..., perhaps we can just set a threshold on the magnitude of the relationship and focus on those above the threshold?, any ideas and further literature on this specific application? $\endgroup$ – elikesprogramming Nov 17 '16 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.