Beginner association question here.
I have a dataset (~2.5M rows) with the following data:
UserID | GroupID | Contract | Contract ID | 2YR Fees Paid? | 5YR Paid? | Total Paid 111 XXX Design AAA Y N 1600 222 YYY Install BBB Y Y 4200 333 ZZZ Design CCC N N 0 444 YYY Install DDD Y N 1600 555 ZZZ Install EEE Y N 1600 111 XXX Install FFF Y Y 1600 222 YYY Install GGG Y N 1600
After grouping and creating a few plots (which show differences between groups) I want to be able to be able to determine:
- Are the different independant variables statistically independated or associated to the dependant variables (fees paid)
- How strong is the association; which columns/features are strongest?
In a perfect world, I would run Pearson’s Correlation to answer these questions; however, since each one of these independent variables are categorical/nominal and not continuous I am using the Chi-Squared Test for Independence and Crammer's V. My crosstabs / contingency tables (chi-sq values simulated):
UserID | count(2 Year Fees Paid) | count(2 Year Fees NOT Paid) 111 750 350 222 80 15 333 580 250 … 10,000 Rows (unique UserIDs) degrees of freedom = 9999 Chi-Squared statistic = 41060.23 pValue = 0.0 Crammer’s V: sqrt(41060.23 /2500000) = 0.128 GroupID | count(2 Year Fees Paid) | count(2 Year Fees NOT Paid) 111 4500 1200 222 2800 900 333 9000 3000 … 450 Rows (unique GroupIDs) degrees of freedom = 449 Chi-Squared statistic = 11441.20 pValue = 0.0 Crammer’s V: sqrt(11441.20 /2500000) = 0.067 Contract Type | count(2 Year Fees Paid) | count(2 Year Fees NOT Paid) Design 350000 146000 Install 250000 149000 ... 2 Rows (unique Contract Types) degrees of freedom = 1 Chi-Squared statistic = 981.50 pValue = 0.0 Crammer’s V: sqrt(981.50 /2500000) = 0.0198
Thus, I performed a Chi-Squared Test for independence (instead of Fisher's Exact Test) on each one of the grouped datasets. After using code and testing out the calculation by hand (to confirm it is correct) I am getting very large Chi-Squared test statistics, which means p-values of essentially 0; strong ability to reject the null hypothesis that the two variables in question are independent and not associated.
Afterwards, I calculated the Crammer’s V to gain some insight into which of the variables are most in question. I am worried because my Chi2 Stats are so huge - is this ok? Am I violating any fundamental assumption for this test? Should I take a random sampling of the data instead of using all values to reduce this? Although I do have a ‘continuous’ variable available (total_paid), I do not believe it is appropriate to use ANOVA. Is my analysis correct and the large test stats simply mean that there is very strong evidence to reject the null hypothesis that the variables are independent?
FYI I am using pyspark and python (pandas, scipy, etc).
Any advice is super appreciated!