Performing one-hot encoding on a very large dataset I am currently analysis a data set containing 654281 observations and 27 variables. I aim to perform binary logistic regression and many of my variables are categorical.
I know one hot encoding is supposed to help with the performance of the classification model but I do not know how many variables to do it on.
Of my 27 variables, 23 are categorical, so in theory I should encode all of them. However, some of them have a ridiculous amount of levels. The levels range from 2 to 80,677, e.g. one being 35, one 738, one 13000. I am not sure if I need to encode them all, as it makes my data set incredibly large with a ridiculous amount of variables. Should I only encode the ones with a small number of levels, if so, what is the cut off point?
Thanks
 A: Just to add to @Tim's answer, other than onehot encoding, there are indeed useful encoding schemes  which do not increase the number of columns at all. Target encoding is already pointed out by @Tim. Other than that, you can try count or catboost encodings. For some explanations on the logic of these encodings, see my post. A code example can be found in my notebook.
A bonus nice thing is all these encoders are already implemented in python package categorical-encoders.
A: It is not that one-hot encoding "helps with performance". One-hot encoding for categorical variables is necessary, at least for algorithms like logistic regression, as you can learn from the Why do we need to dummy code categorical variables thread.
If you have big number of categories, there are some alternatives or ways of making one-hot encodings more managable. If memory usage is the problem, you can use sparse data structure for storing such data. If number of categories is a problem, you can reduce the number of categories by collapsing them, or using hashing trick, i.e. use hash function to "randomly" map them to lower number of categories. Alternatively, you can use target encoding, where you replace the category labels with mean of the target variable.
