I'm working on anomaly detection in CTU-13 dataset. Records are labeled and there are a few categorical features with many categories (for example one of the features "State" has over 250 possible, categorical values - and there are more categorical features!). I could (and probably will, at some point) perform one-hot-encoding, but for now I decided not to, because I would end up with 10 original features, 250 new, sparse features. That's why I decided to pandas.factorize my categorical features and then treat them like numerical ones.
However - sklearn's OSVM works really slowly unless data is standarized. I have already applied QuantileTransformation to my features that are numerical by nature (like "duration", "number of bytes" etc.) and performance of my classifier got drastically better (time of execution was much shorter and overall accuracy of my classification got higher).
Now, I'm aware standarization/normalization these features is wrong, because numbers created through factorization don't represent anything but the category. Regardless, I have already tried using StandardScaler and as I expected, time of execution dropped drastically (we are talking about drop from 3-4h to 20 minutes). When it comes to results... This is interesting. AUC dropped significantly in 4 out of 5 tested scenarios, but other metrics (accuracy, recall, F1 score) spiked up to almost 90 percent (from 70-ish). This suggests there may be a way to somehow optimize my input for good execution time and classification scores, but perhaps it's not as simple as scaling these features.
Articles I found on the internet would factorize these categorical data, but never really did anything more about it, so I had hard time finding a solution other than one-hot-encoding. Should I just jump to it? Is there a better way to factorize data, so it's scaled/standarized already?
