Before the part of the question related to the title, here is some background:
I'm build a model that tries to predict a binary outcome ($Y$), given a set of features. Some of this features are categorical, so I need to create dummy variables for each categorical value. However, in some variables, I get an insane ($>100$) number of categorical values. In order to somehow summarise this, I design the following strategy:
- Select categorical values that represent more than 5% of the entries
- From the remaining, select values that have a different distribution between $Y=0$ and $Y=1$.
This second strategy leads to the crux of the question, ergo, Fisher's exact test. For each categorical value ($X$), I produce a classification table with the rows as $Y=0$ and $Y=1$ and the columns $X=0$ and $X=1$. Then I perform R's fisher exact test. As far as I can tell I'm obeying all the test's assumption and it makes sense.
However, I found some results that are not intuite and I would like to have input on it. As an example, consider the followin classification table
x 1 0 Y 0 0 69377 1 1 2146
This gives a p-value of $0.03002$. Now, I know that this is a marginally significant p-value. I know that, with so many test, one should apply multiple test correction (although for this case, as I'm more worried about Type II errors, maybe I shouldn't). As I understand the reason of the low p-value (we have high confidence on the estimated frequency from $X=0$, and so the observed value of $X=1$ is very off... But man, it's only one value... It would make sense for me to filter out this (as they will be useless in the model), but how can one design a rational cutting line?