0
$\begingroup$

Beginner association question here.

I have a dataset (~2.5M rows) with the following data:

UserID | GroupID | Contract | Contract ID | 2YR Fees Paid? | 5YR Paid? | Total Paid 
111      XXX        Design      AAA           Y                 N           1600
222      YYY        Install     BBB           Y                 Y           4200
333      ZZZ        Design      CCC           N                 N           0
444      YYY        Install     DDD           Y                 N           1600
555      ZZZ        Install     EEE           Y                 N           1600
111      XXX        Install     FFF           Y                 Y           1600
222      YYY        Install     GGG           Y                 N           1600

After grouping and creating a few plots (which show differences between groups) I want to be able to be able to determine:

  1. Are the different independant variables statistically independated or associated to the dependant variables (fees paid)
  2. How strong is the association; which columns/features are strongest?

In a perfect world, I would run Pearson’s Correlation to answer these questions; however, since each one of these independent variables are categorical/nominal and not continuous I am using the Chi-Squared Test for Independence and Crammer's V. My crosstabs / contingency tables (chi-sq values simulated):

UserID | count(2 Year Fees Paid) | count(2 Year Fees NOT Paid)
111         750                       350
222         80                        15
333         580                       250
…
10,000 Rows (unique UserIDs)
degrees of freedom = 9999 
Chi-Squared statistic = 41060.23 
pValue = 0.0 
Crammer’s V: sqrt(41060.23 /2500000) = 0.128


GroupID | count(2 Year Fees Paid) | count(2 Year Fees NOT Paid)
111          4500                      1200
222          2800                      900
333          9000                      3000
…
450 Rows (unique GroupIDs)
degrees of freedom = 449 
Chi-Squared statistic = 11441.20 
pValue = 0.0 
Crammer’s V: sqrt(11441.20 /2500000) = 0.067


Contract Type | count(2 Year Fees Paid) | count(2 Year Fees NOT Paid)
Design             350000                    146000
Install            250000                    149000
...
2 Rows (unique Contract Types)
degrees of freedom = 1 
Chi-Squared statistic = 981.50 
pValue = 0.0 
Crammer’s V: sqrt(981.50 /2500000) = 0.0198

Thus, I performed a Chi-Squared Test for independence (instead of Fisher's Exact Test) on each one of the grouped datasets. After using code and testing out the calculation by hand (to confirm it is correct) I am getting very large Chi-Squared test statistics, which means p-values of essentially 0; strong ability to reject the null hypothesis that the two variables in question are independent and not associated.

Afterwards, I calculated the Crammer’s V to gain some insight into which of the variables are most in question. I am worried because my Chi2 Stats are so huge - is this ok? Am I violating any fundamental assumption for this test? Should I take a random sampling of the data instead of using all values to reduce this? Although I do have a ‘continuous’ variable available (total_paid), I do not believe it is appropriate to use ANOVA. Is my analysis correct and the large test stats simply mean that there is very strong evidence to reject the null hypothesis that the variables are independent?

FYI I am using pyspark and python (pandas, scipy, etc).

Other reasources used: Cooccurence of Two Categorical Variables, Correlations with categorical variables, https://stackoverflow.com/questions/45020538/how-to-apply-cramer-v-on-2x2-matrix

Any advice is super appreciated!

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Chi-square tests are highly sensitive to sample size, big n's produce big stats, a finding that dates back to the earliest literature on discrete analysis and models. That said, your question is a bit unclear or lacking in details. For instance, why (and how) did you collapse the ~2.5m rows down to 10,000 unique ids? Moreover, you appear to have a dependent variable (total paid?), and, assuming total paid is the DV, why you would use two-way contingency tables at all? Please clarify your logic. $\endgroup$
    – user78229
    Commented Dec 1, 2017 at 19:53
  • $\begingroup$ I reduced the ~2.5M rows into 10K unique IDs (and the 450 unique GroupIDs, etc) by using crosstab (pandas or pyspark) and/or sql aggregation functions (groupBy-ID, aggregate-count). This was done to assembly the data into contingencies (count) tables, which I believe are necessary for Chi2 test for independence calculation. DV here is "2YR Fees Paid" (total paid is simply 2Yr*1600+5Yr*3600 - meaning it is essentially categorical as well). My plan is to repeat this process to understand importance and significance of IndVariables in relation to 5Yr Fees next (after 2Yr). $\endgroup$
    – whs2k
    Commented Dec 1, 2017 at 20:41
  • 2
    $\begingroup$ Personally, I don't like your approach at all. You're losing so much information by treating your DV as categorical. It would be so much more informative, not to mention avoiding the issue you're having with inflated chi-square stats, if you were to treat it as a regression model. Perhaps a hierarchical, mixed effects model (e.g., unique ids within group ids), but you would have to share more info about the data structure. $\endgroup$
    – user78229
    Commented Dec 1, 2017 at 23:27
  • $\begingroup$ Good point! I’ve got enough data to seaborn pairplot and fit lines on best combinations of variables to determine associations; why force it? $\endgroup$
    – whs2k
    Commented Dec 2, 2017 at 0:43

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.