I need some pointers / suggestions ... I like to profile a whole bunch of categorical user data - i.e. things like who talks to who, who deals with what product categories typically, and identify ones that don't follow a norm. An anomaly detection problem perhaps ... but what sort of algos should I be looking at? Should I be hot encoding my fields to achieve this?

All the examples I read (k means, one class SVM) are focused on numerical datasets ...

  • 3
    $\begingroup$ What do you mean by "profile" in the second sentence? I think you may be interested in topics like "Contingency tables" and "Chi Squared test" but your question is too vague/imprecise to really know what you want to do. $\endgroup$ – Brad S. Dec 1 '17 at 17:33
  • $\begingroup$ Thx. Profile, I was basically meaning 'learning what normal is'. I.e. statistically this happens reguarlly or not.. $\endgroup$ – Roc Ingersol Dec 1 '17 at 17:42

It depends entirely upon the meaning of your data attributes, which is left out in your question. You should provide more information about the data if you want more detailed help.

Whether a certain encoding "works" or not depends entirely upon the situation at hand, so you will have to reason about what you are trying to find out. You may even have to use different transformations for different attributes.

For example, if there are 10 "product categories" and we encode them into numbers [1, 2, ..., 10], then a numerical anomaly detection method based on Euclidean distance will say that product 1 is "farther" away from product 10 than product 2 is, which may or may not be nonsensical.

There is plenty of reading on categorical clustering available, for example, search for "categorical k-means", and you will probably find problems similar to yours. If you prefer books, Aggarwal's recent book on outlier detection (www.springer.com/us/book/9781461463955) has a chapter on categorical variables.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.