2
$\begingroup$

We're using LightGBM on a dataset with a low signal-to-noise ratio (very easy to overfit, achieving OOS accuracy of 60% is considered a big win) where most features have low predictive power and the overall predictive power of the model comes from its ability to learn complex (nonlinear) interactions between multiple features.

Given that setting high enough values for min_sum_hessian_in_leaf and min_gain_to_split should prevent excessive tree growth/complexity, does it make sense to also optimize num_leaves and max_depth or can we just set them to very high values and trust that if min_sum_hessian_in_leaf and min_gain_to_split are properly tuned, the size/complexity of the trees will be under control? In other words, is there a point in also tuning num_leaves and max_depth?

Update (additional context): It's a classification problem. I'm using Optuna (TPE) for the hyperparameter optimization and KFold CV to calculate the objective function. For each set of hyperparameters, one model is trained for each data fold and their mean validation score (plus a couple of other metrics) is used to select the best sets of hyperparameters.

Thank you.

$\endgroup$
1
  • 1
    $\begingroup$ Nice question (+1). Welcome to SE.CV. I think what you are doing is fine so far and there are one points worth focusing. Please see my answer below where I expand on this comment. $\endgroup$
    – usεr11852
    Mar 7 '21 at 17:34
2
$\begingroup$

No. We should focus our attention elsewhere, in particular to the validation approach we use.

In a situation that SNR is low it is especially pertinent to focus our attention to the validation schema in place. It is easy to be lulled in a false sense of security because we unintentionally over-fit our data or (less often) under-fit our data because we have regularised too aggressively without realising it.

It is good practice to use a repeated cross-validation schema (e.g. simple repeated CV) as well as to have a hold-out sample to ensure that our error metrics (e.g. RMSE and MAE) are comparable. If the two metrics are not generally comparable it might suggest that our algorithm potentially has poor generalisation performance despite our extensive resampling. Similarly, it is important to anticipate potential clusters in our data (e.g. spatial or time-associations) that might make a simple resampling technique insufficient in terms of giving us "independent" sub-sets. An example from a past project I worked: we were performing a learning task where the unit of analysis was a residential flat; nevertheless, flats within the same building (i.e. block of flats) behaved so similarly that while our unit of analysis as a flat indeed, we were inadvertently leaking information. As soon as, we used building as the unit of resampling, the ability of our learner to generalise as well as its interpretability went up despite our CV metrics going nominally down.

Touching on the "under-fitting" too: As we increase the number of hyperparameters to optimise against we increase our search space. That means that our grid- or random-search needs to explore a wide search space so we have an adequately performant combination. The difficulty of optimization increases roughly exponentially with regard to the number of parameters; i.e. we need to exponentially increase the number of necessary trials when we increase the number of parameters. If two hyper-parameters have "roughly the same" influence then optimising both may not be the best use of our time. (Yes, their are some rule-of-thumbs that can be used (e.g. 60 trials in RS) but they not silver bullets nor give us strong guarantees that we haven't under-fitted.)

Please note that not all hyper-parameters matter equally. There is some work on this matter (e.g. see van Rijn & Hutter (2018) Hyperparameter Importance Across Datasets and references within). If one is willing to do a reasonable large exploration it might be worth looking at such work in more detail.

To conclude: Invest on the validation schema. Given we have a reasonable CV schema in place optimising for many additional hyper-parameters starts having diminishing returns. In general, I would always optimise for max_depth because it has the most profound impact on the quality of fit and the complexity of the learner before any of the other three hyperparameters mentioned. That said, given you use min_sum_hessian_in_leaf and min_gain_to_split I think you won't gain a lot more by optimising for max_depth too (assuming that max_depth is "deep-enough" already). Focus attention on getting better features to get better performance.

$\endgroup$
4
  • $\begingroup$ Thank you very much for the thorough answer. I've found that min_sum_hessian_in_leaf and min_gain_to_split are very important for this dataset. Without them, the model overfits very badly if max_depth is high enough to learn the interactions we need. If max_depth is low enough to mitigate some of the overfitting, then the model seems not to learn those interactions. I've updated my question to provide additional info. Just to be sure I understood, given that I'm already optimizing the two above parameters and using proper validation, is your recommendation not to optimize max_depth? $\endgroup$
    – BrunoF
    Mar 7 '21 at 22:22
  • 1
    $\begingroup$ I am glad I could help. Thank you for this additional information. Yes, if your validation schema is robust enough I don't think one would be expecting big gains from adding max_depth (or num_leaves) too given this additional information. (I will edit my answer to explicitly say that.) Optuna is very good, I use it too. :) (Also you don't mention what metric you are using exactly but you do mention Accuracy; Accuracy in itself ($\frac{TP+TN}{N}$) is usually a suboptimal classification performance metric (e.g. see stats.stackexchange.com/questions/312780). $\endgroup$
    – usεr11852
    Mar 7 '21 at 23:42
  • $\begingroup$ Thank you for the edit, it's 100% clear now :). About the metric, I've mentioned accuracy because for most people it's easier to think in terms of accuracy. I use Matthews correlation coefficient (MCC). Anyway, thanks for also pointing that out. $\endgroup$
    – BrunoF
    Mar 8 '21 at 15:14
  • $\begingroup$ On an unrelated note, today I'm working on a different problem involving resampling time-series data with a high serial correlation. Somehow I've remembered your example of the similar flats and it helped me understand an issue that caused inflated validation scores. Thanks again :) $\endgroup$
    – BrunoF
    Mar 23 '21 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.