# Test statistical significance with small number - Fisher's exact test

Before the part of the question related to the title, here is some background:

I'm build a model that tries to predict a binary outcome ($$Y$$), given a set of features. Some of this features are categorical, so I need to create dummy variables for each categorical value. However, in some variables, I get an insane ($$>100$$) number of categorical values. In order to somehow summarise this, I design the following strategy:

• Select categorical values that represent more than 5% of the entries
• From the remaining, select values that have a different distribution between $$Y=0$$ and $$Y=1$$.

This second strategy leads to the crux of the question, ergo, Fisher's exact test. For each categorical value ($$X$$), I produce a classification table with the rows as $$Y=0$$ and $$Y=1$$ and the columns $$X=0$$ and $$X=1$$. Then I perform R's fisher exact test. As far as I can tell I'm obeying all the test's assumption and it makes sense.

However, I found some results that are not intuite and I would like to have input on it. As an example, consider the followin classification table

                 x
1      0
Y    0     0    69377
1     1     2146


This gives a p-value of $$0.03002$$. Now, I know that this is a marginally significant p-value. I know that, with so many test, one should apply multiple test correction (although for this case, as I'm more worried about Type II errors, maybe I shouldn't). As I understand the reason of the low p-value (we have high confidence on the estimated frequency from $$X=0$$, and so the observed value of $$X=1$$ is very off... But man, it's only one value... It would make sense for me to filter out this (as they will be useless in the model), but how can one design a rational cutting line?

## 2 Answers

Why do you care about statistical significance, if your goal appears to be a predictive model? The two things typically have very little to do with each other.

You should consider to simply test the different versions on your training data using cross-validation and pick the summarization that leads to the best (or perhaps fewer categories than in the best, in order to not overfit too much) average performance across cross-validation folds?

• There will be a step afterwards where we will do that. However, given the structure of the code and the pipeline, at this stage (data acquisition and preprocessing) and can not do a proper train and test strategy. I would like to really on relatively simple "tests" to measure the potential interest of each categorical values. I can, of course, just add them all and do the removal at the modelling level. But I thought it could be interesting the trim down the number of columns a bit up front. – Diogo Santos Mar 8 at 8:55
• Sure, just statistical significance is not really that good a criterion. – Björn Mar 8 at 9:30
• So, what would you suggest, besides doing the actual model? – Diogo Santos Mar 8 at 9:51

Alternatively, you can map the categories to the observed probability of Y in each category (in the training data). This will create one numerical variable instead of a large number of dummy variables. You can still treat the rare categories differently by combining them.

Here is a blog post that describes this approach in detail: http://www.win-vector.com/blog/2012/07/modeling-trick-impact-coding-of-categorical-variables-with-many-levels/