Categorical data representation in a supervised learning setting Given a dataset X (all features are categorical) and corresponding label y I need to find some meaningful representations of these categorical features. Are there any methods for that? I assume that they should be based on some existing learning algorithms.
 A: There are several approaches. Here are some to give you an idea:
Labels
Just count the occurring classes and assign them unique values, e.g. if you have to classes cat and dog, assign a 0 to cat and 1 to dog.
One-Hot-Vector
One approach to this is called One-Hot-Vector. The idea is to find unique values in your data and then encode them in vector containing 1 at the specified slot, 0 else. 
Let's say you have a column containing color data such as
['red', 'green', 'yellow', 'red', 'green']

for your given samples.
The One-Hot-Vector would be as follows:
red green yellow
----------------
1    0    0
0    1    0
0    0    1

The reason behind this is that if you were to simply use numerical values, you would imply some kind of relation between the values. One-Hot-Vectors avoid this.
So if your data was
point color label
----------------
x_1 'green' 'cat'
x_2 'red' 'dog'
x_3 'yellow' 'cat'

it becomes
point color1 color2 color3 label
--------------------------------
x_1   0      1      0       0
x_2   1      0      0       1
x_3   0      0      0       0

which can then be applied to any learning algorithm that expects numeric data points.
Binary Encoding
You can find a set of unique values and encode them binary. Using the encoded values you can e.g. employ kmeans and use the Hamming Distance as a metric.
Bag of Words
Find unique values and put them in one set per data point. Disregard order, but keep count.
Decision Trees
Can handle categorical data themselves.
Association Rules
Some models work with rules, e.g. 

if a customer bought beer, he will most likely also buy chips

Histograms
Find unique value per data point and then count them. The resulting histogram can then be used as a feature vector itself.
