I have a question regarding how I should convert categorical data to numerical data. I'm using this kdd99cup intrusion detection dataset, which has a 41 attributes and class label is the type of attacks. I need to feed this dataset to self-organizing map because I want to gauge how difficult it is for an unsupervised algorithm to group data of the same class together. The problem is some of the columns have categorical data but if I use one-hot encoding, the number of dimensions increases to around 120 and I heard that SOM does not work well if the dimension is too large. How should I deal with these categorical attributes?
There are many possible solutions if you have many features in the data:
- feature selection -- just drop some of the categories that are of less importance, for example, in a natural language processing scenario, this may be some very rare words, that are unlikely to be useful, though in most cases this would be the worst possible solution (e.g. the rare words can have great predictive power),
- dimensionality reduction -- use a dimensionality reduction algorithm (for example, PCA, or have neural network produce categorical embeddings) on the data,
- regularization -- I have no experience with using self-organizing maps, but most of the machine learning algorithms have some ways of regularizing them (see regularization), e.g. by using $L_1$ or $L_2$ penalties, dropout, etc,
- hashing trick -- you can reduce the number of dimensions by randomly binning together some of the categories, it sounds crazy, but it is known to work quite well in some cases.