0
$\begingroup$

I realize that some derivative of this question has been asked here before, but none have addressed the situation where there is ONLY high-cardinality, categorical data, and that the labels themselves are not important, but the combinations of labels are.

The data will often look something like this (in this case, log data from a firewall)

timestamp          source ip      dest ip    dest url          categories
12:00:00.000       4.4.4.4        1.2.3.4    www.badguy.com    malicious, shopping, security
12:00:00.001       3.3.3.3        6.7.8.9    www.badguy.com    malicious, shopping, security

Ultimately, I am trying to classify the billions of lines of nominal features into clusters of arbitrary 'personas' that we can then use to help detect anomalous behavior and/or predict which 'persona' a particular behavior could be attributed to. About the only feature that could possibly be OHE is 'categories', but it gets ugly very fast, since we have trillions of possible combinations of categories to IPs to urls.

I have considered just calculating the probability of an exact combination of features occurs and using that as a feature (maybe with weighting based on domain knowledge), but I don't see how that could serve as anything meaningful in a classification algorithm.

Does anyone have any suggestions for encoding feature combinations for use in a simple clustering model?

$\endgroup$
  • $\begingroup$ I should also mention that I have looked at this fantastic article, but again it doesn't apply. We aren't trying to predict income (numeric) from zip codes (nominal, low cardinality), for instance. It's more like we're trying to cluster zip codes, first names and house color into their natural groupings, but there are millions of zip codes, first names, and house colors. $\endgroup$ – TheProletariat Apr 26 '17 at 22:39
  • 1
    $\begingroup$ You need to perform good feature extraction. Too much noise, too little signal. $\endgroup$ – Anony-Mousse Apr 27 '17 at 7:46
  • $\begingroup$ See also stats.stackexchange.com/questions/146907/… $\endgroup$ – kjetil b halvorsen Jun 25 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.