I have a large dataset (approx. 400,000 rows, 650 columns). Out of this 650 columns, 500 are dense (nonzero over 10%) and 150 are sparse (nonzero less than 10%).

As expected, the random forest has difficulty selecting the sparse features for any splits, while they are expected to hold at least some predictive power (A forest with only these features can obtain a gini of 0.2).

My initial thought was to apply another algorithm than a random forest, but e.g. a neural network seems to perform worse than a forest based algorithm here. (I am still considering to try other models, but curious if I can tackle this in some way with a RF.)

A second thought was to apply PCA to the sparse features to obtain a smaller set of features, and include those together with the dense features in a random forest. However, because of the large amount of zero's, one component already captures >99.99% of the variance.

Does anyone have other suggestions for approaching this issue?



  • 1
    $\begingroup$ Does the single component improve the performance of the RF model? What about including the other components? $\endgroup$
    – Sycorax
    Jan 6, 2018 at 16:23
  • $\begingroup$ For linear models it is suggested to use the mean of the column to replace the "na" because it doesn't change the column parameter estimate (much?) and it allows the non-NA columns to add their informative contribution. Have you considered something like the R "mice" package, and missing variable imputation? $\endgroup$ Jan 8, 2018 at 1:43
  • $\begingroup$ @Sycorax. I looked into it again and PCA had issues because I did not scale the data beforehand. After scaling, 19 features contained 95% of the variance of the sparse data. Sadly, it did not improve model performance. $\endgroup$
    – Florian
    Jan 8, 2018 at 8:44

1 Answer 1


Why don't you try running a separate model on sparse data? You're already running an ensemble method, so it shouldn't be a problem to incorporate another model.

For such data linear models with sparsity-promoting regularization (like LASSO or Elasticnet) can be a valid alternative for tree models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.