I'm working on anomaly detection in CTU-13 dataset. Records are labeled and there are a few categorical features with many categories (for example one of the features "State" has over 250 possible, categorical values - and there are more categorical features!). I could (and probably will, at some point) perform one-hot-encoding, but for now I decided not to, because I would end up with 10 original features, 250 new, sparse features. That's why I decided to pandas.factorize my categorical features and then treat them like numerical ones.

However - sklearn's OSVM works really slowly unless data is standarized. I have already applied QuantileTransformation to my features that are numerical by nature (like "duration", "number of bytes" etc.) and performance of my classifier got drastically better (time of execution was much shorter and overall accuracy of my classification got higher).

Now, I'm aware standarization/normalization these features is wrong, because numbers created through factorization don't represent anything but the category. Regardless, I have already tried using StandardScaler and as I expected, time of execution dropped drastically (we are talking about drop from 3-4h to 20 minutes). When it comes to results... This is interesting. AUC dropped significantly in 4 out of 5 tested scenarios, but other metrics (accuracy, recall, F1 score) spiked up to almost 90 percent (from 70-ish). This suggests there may be a way to somehow optimize my input for good execution time and classification scores, but perhaps it's not as simple as scaling these features.

Articles I found on the internet would factorize these categorical data, but never really did anything more about it, so I had hard time finding a solution other than one-hot-encoding. Should I just jump to it? Is there a better way to factorize data, so it's scaled/standarized already?


1 Answer 1


Am I correct that pandas.factorize only recodes the categorical variable as numerical, with numbers as labels, so it is just a numerical variable? That is not the way to treat a categorical variable.

There is no reason to be afraid of the 250 new columns, just use sparse matrix methods. And look at the ideas in Principled way of collapsing categorical variables with many levels?, especially the fused lasso could be helpful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.