On the BorutaPy GitHub page, the authors state: "We highly recommend using pruned trees with a depth between 3-7" (https://github.com/scikit-learn-contrib/boruta_py). What justification is there for using a depth in this range? I've seen this recommendation reiterated many times by others citing the BorutaPy authors without explanation. If I have many features and suspect that there are highly complex patterns in my dataset, would you recommend deviating from this suggestion and using a larger depth?
- Reading the Boruta paper ("Feature Selection with the Boruta Package " (2010) by Kursa & Rudnicki) there is no depth recommendation.
- Checking the
borutacodebase in version 1.3 (the one from the publication) and a recent one (8.0) there is no explicit setting of tree depth. For that matter I have the suspicion that
randomForest::randomForestdoesn't allow that parameter to be set. We can set
maxnodesso those will probably have a similar effect but not really depth.
- Almost certainly this is to do with computational speed.
borutacan be very computationally expensive. One way to save time is to grow shallower trees. It is true for all ensemble methods using trees as base learners and it will be true here too.
- Treat tree depth as a hyper-parameter. This will lend good credibility to the subsequent analysis. Show that for 4, 8 and 12 for example we get the same results (qualitatively). That can go to the Appendix, but actually nullifies any critique that what we see is an artefact of the depth used. If you see differences investigate, report and comment on them, it will only add points to your work as it will show there has been an informed application of the B0ruta procedure rather than a "quick rinse". Similarly, if massive difference are observed, then we know something is fishy and probably we should think twice about using Boruta in the first place.
- We do not have a view of why Boruta needs to be used but employing penalised algorithms might be preferable to performing feature selection early on in an analysis. (Or we might have modelling constraints we have to adhere in which case disregard this point)
It's a heuristic algorithm, based on a clever idea and backed up by some empirical results, but as such, it does not give you any guarantees of correctness. The recommendation most likely is based on empirical observations. So yes, it can be the case that different hyperparameters may behave differently for different cases.