1
$\begingroup$

I am currently analysis a data set containing 654281 observations and 27 variables. I aim to perform binary logistic regression and many of my variables are categorical.

I know one hot encoding is supposed to help with the performance of the classification model but I do not know how many variables to do it on.

Of my 27 variables, 23 are categorical, so in theory I should encode all of them. However, some of them have a ridiculous amount of levels. The levels range from 2 to 80,677, e.g. one being 35, one 738, one 13000. I am not sure if I need to encode them all, as it makes my data set incredibly large with a ridiculous amount of variables. Should I only encode the ones with a small number of levels, if so, what is the cut off point?

Thanks

Improve this question
$\endgroup$
3
  • $\begingroup$ What to you mean by "help with the performance"? What other alternative are you considering to one-hot encoding for categorical variables? $\endgroup$
    – Tim
    Commented Jun 29, 2020 at 13:33
  • $\begingroup$ What kind of data is this (how was it created ?). Can you give an example of a categorical variable with a huge number of levels? Also the first sentence says 2 variables, I assume that is a typo? $\endgroup$
    – Robert Long
    Commented Jun 29, 2020 at 17:39
  • $\begingroup$ By 'help with performance' I meant it improves the predictive power of the classification model, I'm not really considering other methods, as my client asked me to research and implement one-hot encoding, but I can use another method if it is more appropriate. The data is fraudulent card transactions and I have corrected the mistake. Also, some examples of my variables include "currency used for the transactions", which has 22 levels, "product code", which has 75 levels and "product code non fuel", which has 738 levels. $\endgroup$
    – Sydney
    Commented Jun 30, 2020 at 8:29

2 Answers 2

Reset to default
2
$\begingroup$

Just to add to @Tim's answer, other than onehot encoding, there are indeed useful encoding schemes which do not increase the number of columns at all. Target encoding is already pointed out by @Tim. Other than that, you can try count or catboost encodings. For some explanations on the logic of these encodings, see my post. A code example can be found in my notebook.

A bonus nice thing is all these encoders are already implemented in python package categorical-encoders.

Improve this answer
$\endgroup$
1
$\begingroup$

It is not that one-hot encoding "helps with performance". One-hot encoding for categorical variables is necessary, at least for algorithms like logistic regression, as you can learn from the Why do we need to dummy code categorical variables thread.

If you have big number of categories, there are some alternatives or ways of making one-hot encodings more managable. If memory usage is the problem, you can use sparse data structure for storing such data. If number of categories is a problem, you can reduce the number of categories by collapsing them, or using hashing trick, i.e. use hash function to "randomly" map them to lower number of categories. Alternatively, you can use target encoding, where you replace the category labels with mean of the target variable.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.