0
$\begingroup$

I have a dataset of 1000+ records. In this dataset, I have two categorical features, Ticket-Label and Survived. Ticket-Label has 54 unique categories and Survived has 2 categories. The chi square test for the features, gives a chi value of 766.59 with a p-value of 0.011. So according to test the two features are dependent but is it so? Could the test be wrong? I am a little skeptical since the number of categories are high for Ticket-Label

$\endgroup$
0
$\begingroup$

Chi-square tests in the presence of both a large amount of data and many possible categories or levels are notoriously unreliable. Large chi-square values can be inflated due to, e.g., cell frequencies in the two-way cross-tab that are small (or zero) relative to expected counts and the test's extreme sensitivity to sample size.

A simpler test on your raw data before collapsing levels might be to run an ANOVA and use the resulting F-test as a directional indicator for dependence. First, transform Survived into a zero-one dummy variable and use it as the target or dependent variable. A multiple group comparison test such as Student-Newman-Keuls might be useful in gaining insight into contrasts between the 54 levels of Ticket-Label.

You could also run a logistic regression and examine the odds-ratios.

If one can motivate it and it doesn't negatively impact any downstream analyses, collapsing redundant levels of a categorical variable is rarely a bad idea.

$\endgroup$
1
$\begingroup$

This p-value represents the probability that ALL 54 categories have the same average survival rate vs there is at least one category that has a different one. As a general rule, you are right, the more categories you have the more likely to have one with at least different average survival rate even by pure chance! This is logical.

Is there a way that you could maybe merge some of these categories together? That could make the test a bit more reliable

$\endgroup$
  • $\begingroup$ Thank you for the advice @Vasilis. I'll try merging some of the categories. :) $\endgroup$ – Rishi Sharma Jan 20 '19 at 17:25
  • 1
    $\begingroup$ This answer seems to state that the chi-square is more likely to reject even when the null is true simply by increasing the number of categories. I don't think this is the case; as long as the counts are large enough that the chi-squared approximation is suitable, the overall type I error rate should be approximately the nominal rate. [If the number of categories is such that there are many small expected values, there may be an issue, but it's a problem of small expecteds rather than inherently in the number of categories] $\endgroup$ – Glen_b -Reinstate Monica Jan 21 '19 at 3:20
  • $\begingroup$ I used the expression “as a general rule” without saying anything about the large enough counts. As the number of categories d tends to number of observations n then the observations per category n/d tends to 1 and so smaller fluctuations could make the null hypothesis to be rejected and in real world applications, the assumptions of normality could be more questionable. The OP said 1000+ observations and 54 categories which translates to ~20 observations per category and to my mind at least, this is not the case of large enough counts per category. $\endgroup$ – Vasilis Vasileiou Jan 21 '19 at 6:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.