Is it a valid technique to exclude features based on occuring very rarely? Or would it be better to do feature selection on subsets?

Question

I have a large sparse binary dataframe with codes from health data.

• The train columns are one-hot-encoded ICD Codes, OPS Codes, ATC Codes, and other covariates such as age etc. (scaled to be between 0, 1)

• My y values are binary: has a stationary hospital case in the following year for each insurant

• The training dataset has 162153 datapoints, the test dataset has 69495 datapoints.

The code occurance is extremely sparse, 11861 out of the total 26315 codes (across all code types) occur 3 times or less in the entire train set.

I want to use logistic regression with lasso for selecting the top features. However, having all codes in the dataframe, this takes a long time to run.

1. Would it be a valid approach to discard all codes which occur 3 times or less? It makes sense because they wouldnt be very predictive overall, and could be predictive due to randomness only if they occur so rarely. However, I couldn't find any information on that technique online.

2. My other idea would be to use logistic regression with lasso feature selection for each code type (ATC, ICD) individually, then put the selected codes into one dataframe and do the code selection again. Would this be a good approach?

(I thought about using PCA, however it is not applicable in this project since I need to provide a list of selected Codes used in the model which are important for prediction. The goal is still to predict whether someone will be in the hospital, but they want to know which codes are relevant as a side information)

Answer 1

It's a reasonable approach to exclude features that are very rare, but it's neither guaranteed to be a good idea or to result in the best model. Exactly how rare features need to be is another question, which could be a hyperparameter of the whole training pipeline that one could select based on some suitable evaluation approach (e.g. what optimizes AUC in cross-validation grouped by patient).

There's quite a few (potentially good or bad) alternatives:

• Grouping codes: If lots of rare ICD for e.g. some rare autoimmune disorders occur in very few patients, maybe it makes medically sense to group them - either in a "other autoimmune disorders" category or grouping them with similar conditions that are more common. To some extent this should be based on medical knowledge, or one could to some extent rely on the hierarchies in the coding system. Perhaps it's not even sensible to always keep the lowest available code level and it could be better to combine codes at a slightly higher level.
• Univariate/lower dimensional screening (your idea #2): This is generally a problematic approach, because features may be selected that are not so useful (e.g. some features are no longer useful once one takes other features into account), and features that are useful may not be selected (because you need some other features in the model to recognize that they are useful). Additionally, unless you put this whole selection procedure into your evaluation scheme (cross-validation, bootstrapping or whatever you are using), you will not correctly evaluate the (un)stability and performance of doing this. That this approach is a good idea is "Myth #2" in this article.
• Using a lower dimensional representation: You mention PCA, which may indeed not be the most obvious approach. A more obvious approach would be to use some meaningful embedding (sure, some of these - just like PCA - will make interpretation a bit harder, but there are various explainability methods that might help), which is often worth considering when you have extremely high dimensional categorical features:
  • Option 1 collaborative filtering: Constructs embeddings for patients and any codes you have (like ICD and ATC codes at the same time), which reflect which codes tend to co-occur and what patients have a propensity to have each. For an introduction to this approach, have a look at Chapter 8 of Deep Learning for Coders with fastai & PyTorch. You could then use the patient embeddings as an input to logistic regression (plus maybe still some key codes identified by your medical colleagues as especially important). The nice thing about this approach is that it's unsupervised (in the sense that it does not use the outcome variable you want to model), the bad thing is that it's unsupervised (i.e. the representation of each patient that you get is also not optimized towards predicting the outcome you want to mdoel).
  • Option 2 embeddings based on co-occurrence: Kind of similar to Option 1, but you would not necessarily have to even create embeddings for patients and afterwards use the embeddings for the ICD or ATC codes. The difficult bit here is how to decide how one combines them when multiple ones apply for a patient (it's a lot easier when just a single one applies to a patient). E.g. is just taking the mean sensible?! Or the sum? That makes the approach more difficult.
  • Option 3 switch to using a neural network: The problem of how to combine the embeddings for different codes that apply to a patient are sort of automatically solved, if you use a neural network, such as a transformer architecture (can be used both if the codes are not ordered or if they are) or if there is an order (e.g. how long ago these codes were assigned) some kind of recurrent neural network (such as a LSTM). E.g. with a transformer neural network, the different codes that apply to a patient would be fed into the neural network through and embedding layer and then an attention mechanism "decides" which of these should be given weight.
  • Option 4 random effects: Maybe I should have mentioned this one first, because of it's simplicity. A random effect is kind of a 1-dimensional embedding, i.e. each category just gets represented on a single dimension from $(-\infty, \infty)$. If only one category applies to a patient, this is the classic random effects approach supported by many software packages. What's more difficult here is that multiple categories can apply to a patient. For that, a multi-membership random effects model is needed. I'm not sure whether there's really good options for that kind of model with this type of huge dataset (weirdly, the "more complicated" neural network options are usually quite good with large data, if you only have enough CPUs and GPUs). Using such a low dimensional embedding definitely looses some information that could be preserved by a higher dimensional embedding, but it on the other hand helps with interpretability.
• Target encoding: Sort of a simple poor man's version of embedding option 4 above and many of the same considerations apply. For more on the various options for implementing this see e.g. this recent publication, although the technique is much older (they quote something from 2001 as the first formal description of the technique). The most difficult challenge here is how to avoid overfitting and the paper discusses that. As before, the other question is how to combine multiple categories per patient, as before maybe taking the sum might be okay.

It's also very much worthwhile to read regression modeling strategies on model selection (and also evaluation). Plus, of course a book like Clinical Prediction Models.