I have a dataset listing the software installed for each user. This dataset shall be used (in conjuction with other user datasets) to classify the user into 4 (imbalanced) categories.

There are over 1000 different pieces of software. Each user can have between 15 and 40 pieces of software installed. In fact, the same software can be counted as installed multiple times (I presume this refers to different versions/upgrades of the same software but I'm not sure).

The naive approach (A) would be to use one-hot/dummy encoding for a variable "Software" but this would produce over 1000 features.

A slightly more sophisticated approach would use some sort of feature selection (e.g. using chi-square/mutual information or some wrapper model) and ignore the less informative pieces of software.

And a third one would be to use target/mean encoding.

Which of these would you recommend?

My preference is for target/mean encoding but I wonder how to approach it.

I was thinking initially to just sum up the raw counts for each target value: I'd create 4 features each one corresponding to one of the values of the dependent variable and for each of these I'd count the times the particular software was installed and then sum up over all software installed by the user. The problem is that in this way some popular software (e.g. Word or a browser) will likely "crowd-out" some less commonly used (but potentially more informative) pieces of software. So next I thought that I could replace the raw counts with the percentage. This would give equal weight to each piece of software. But is this OK?

I guess I am probably not the first person to face such a problem. What would constitute a principled/established approach of dealing with such variable number of high cardinality categorical features?

  • 1
    $\begingroup$ You could group all software into N groups and then use the number of apps within a group as a feature. I recommend using domain knowledge. $\endgroup$ Commented Nov 28, 2018 at 14:05
  • 1
    $\begingroup$ Non-negative matrix factorization of the dummy-encoded data would give natural groupings of software that goes together in your user base. $\endgroup$
    – Sycorax
    Commented Aug 1, 2020 at 16:13

1 Answer 1


You did not specify a target variable to predict/classify. If you have that I would look into the (fused) lasso, see Principled way of collapsing categorical variables with many levels?. See also Is the LASSO really applicable for binary classification problems?, another idea in Strange encoding for categorical features.

Without a target variable, this is more like clustering.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.