Improve XGBoost performance in a huge dataset with a lot of missing values

Question

I have a dataset with around 250 features and 4 Millions samples and we obtained a model with Xgboost that has acceptable performance. The dataset has a high percentage of missing value, for an important number of variables. I aim to improve model's performance. As there is a big number of features, I though first to apply some techniques of feature selection, such embedded methods with random forest. But if the percentage of missing values is not a problem for Xgboost, I know that for other algorithms it's not well.

How can I do? Are there any techniques to select features even when missing values are present? Or do I have to apply some techniques to handle missing values first, and which ones? (the option of drop columns is not possible).

It's a classification problem with two very imbalanced classes (predict credit default).

Thanks so much for your help :)

Answer 1

With XGBoost, you already have the feature importance and ability of handling missing values.

For handling missing values, the original paper explains well:

It is important to make the algorithm aware of the sparsity pattern in the data. In order to do so, we propose to add a default direction in each tree node. When a value is missing in the sparse matrix, the instance is classified into the default direction. There are two choices of default direction in each branch. The optimal default directions are learnt from the data... The key improvement is to only visit the non-missing entries. The presented algorithm treats the non-presence as a missing value and learns the best direction to handle missing values.

And, if the missing percentage of a feature is too high to provide any valuable information, why not drop it? If you want to manually impute the missing data, this is very much problem dependent, like missing patterns, data distributions...

For feature importance, you have a tutorial from the official doc.

250 features VS 4 million of samples sounds a good deal, even if imbalance exists. If necessary, you can try either upsampling (random repeat, SMOTE, ADASYN...) the minor class or downsampling the major class.