4
$\begingroup$

I have hundreds of binary features, resulting in a large binary design matrix (though note that my response variable is not binary). I've tried typical models like logistic regression, KNN, and SVMs with specialized kernels (like the Hamming kernel, mentioned here). I've also tried reducing the dimensionality of the data using PCA, though whether PCA is valid for binary data is debated (for example here and here).

None of these approaches have got me very far. Are there other models, dimensionality reduction techniques, feature engineering/selection techniques, etc that are well suited to a problem with hundreds of binary features?

Edit: these binary features were produced from one-hot encoding categorical features with many categories. I first tried encoding these features with integers but that didn't get me far either, probably because the categories are not ordinal.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ What sort of model are you trying to build? What is your objective? If your response variable isn't binary, why are you considering logistic regression? How much data, relative to the number of features, do you have? $\endgroup$
    – jbowman
    Jan 2 at 16:31
  • $\begingroup$ @jbowman It's a multiclass classification problem, I have about 100 times more data than features, would that be enough do you think? $\endgroup$
    – xojfqa
    Jan 2 at 17:21
  • $\begingroup$ Yes, that should be more than sufficient. $\endgroup$
    – jbowman
    Jan 2 at 17:38

1 Answer 1

Reset to default
5
$\begingroup$

I have found that is very useful for large numbers of binary features. In my experience, it tends to create a small number of dense feature vectors that are interpretable as plausible combinations of one-hot features that tend to go together. But this assumes that your hundreds of binary columns are the result of using one-hot or dummy encoding for several categorical variables.

Entity embeddings could also be useful, if you (1) want to use a neural network and (2) have several high-cardinality categorical features to encode.

If all of your columns encode a single categorical variable, then there's really not much to do: most models will just estimate a constant for each level of the category. (A corner case here is a model that uses shallow decision trees. Shallow trees won't be able to split the categories to purity, so some categories will be grouped together.)

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks, you're right that these are from one-hot encoding multiple categorical variables. But could you explain why if they were from a single categorical variable, there would not be much we could do? I don't quite follow what you said about a constant being estimated for each category. $\endgroup$
    – xojfqa
    Jan 2 at 17:22
  • 1
    $\begingroup$ If you only have a single categorical feature, then there's only one nonzero element in the design matrix for each observation. Think about OLS: the intercept is the average value when all other features are 0. With 100s of 1-hot binary features and no intercept, the single 1 in a row means you're computing an average response for that categorical level when all other features are 0. $\endgroup$
    – Sycorax ♦
    Jan 2 at 17:28
  • $\begingroup$ Thanks. Could you give a quick rundown on how NMF is used for dimensionality reduction? I've found plenty of explanations on how the math works, but have not been able to find good references on how to interpret the resulting matrices as new features. With V=WH, the left matrix seems to be the "feature matrix" and the right matrix the "coefficient matrix". Does this mean I take the left matrix's columns as my new features, and ignore the right matrix? $\endgroup$
    – xojfqa
    Jan 2 at 21:04
  • $\begingroup$ The NNMF task is $X \approx WH$, so if $X$ is $n \times p$ for $n$ observations and $p$ features, and you're reducing dimensionality to $k < p$, then it must be the case that $W$ has shape $n \times k$ and $H$ has shape $k \times p$ by the rules of matrix arithmetic. You won't want to discard $H$ because when you apply the NNMF model to new data (at test time or in production), then you'll need to estimate $W$ for the new data given the fixed $H$. $\endgroup$
    – Sycorax ♦
    Jan 2 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.