I have a dataset of 1000+ records. In this dataset, I have two categorical features, Ticket-Label and Survived. Ticket-Label has 54 unique categories and Survived has 2 categories. The chi square test for the features, gives a chi value of 766.59 with a p-value of 0.011. So according to test the two features are dependent but is it so? Could the test be wrong? I am a little skeptical since the number of categories are high for Ticket-Label
Chi-square tests in the presence of both a large amount of data and many possible categories or levels are notoriously unreliable. Large chi-square values can be inflated due to, e.g., cell frequencies in the two-way cross-tab that are small (or zero) relative to expected counts and the test's extreme sensitivity to sample size.
A simpler test on your raw data before collapsing levels might be to run an ANOVA and use the resulting F-test as a directional indicator for dependence. First, transform Survived into a zero-one dummy variable and use it as the target or dependent variable. A multiple group comparison test such as Student-Newman-Keuls might be useful in gaining insight into contrasts between the 54 levels of Ticket-Label.
You could also run a logistic regression and examine the odds-ratios.
If one can motivate it and it doesn't negatively impact any downstream analyses, collapsing redundant levels of a categorical variable is rarely a bad idea.
This p-value represents the probability that ALL 54 categories have the same average survival rate vs there is at least one category that has a different one. As a general rule, you are right, the more categories you have the more likely to have one with at least different average survival rate even by pure chance! This is logical.
Is there a way that you could maybe merge some of these categories together? That could make the test a bit more reliable