0
$\begingroup$

I have a data set (time-series) with the shape {$2190$x$63$}. There are 63 variables, 2 products ($A$ and $B$) worth of 3 years of daily data, thus I have $1095$ observations per product and total of $2190$ observations.

Important note: Product $A$ has spikes every 30th of December.

I've built a lightgbm model which overall works just fine. I've chosen lightgbm because the number of products will be much larger, increasing the size of the dataset. Let's say $20$ products, $1095$ observations per product, $63$ variables, which totals $20*1095=21900$ rows and $63$ columns.

My problem: min_data_in_leaf affects how well spikes of the product $A$ are forecasted.

As far as I can tell, when you have a relatively large dataset {$21900$x$63$} (it can go WELL beyond it), min_data_in_leaf should be large enough to make sure that your model is not overfitting. The problem is, no matter how many products will be included (expanding my data), large min_data_in_leaf will fail to capture these spikes because to distinguish them from other observations I need to specify a small value of min_data_in_leaf. At least to my understanding.

My thoughts on it:

  1. I should stick to tuning other parameters that control overfitting and keep min_data_in_leaf fairly low;
  2. Maybe I need to do more feature engineering? (I've tried, unsuccessfully)

Any help is greatly appreciated!

Visualisation. 1 year of data (I have 3) + forecasts:

Comparing models

My parameters (early stopping is on):

lgb_params = {'boosting_type': 'gbdt',
              'objective': 'regression',
              'metric': 'rmse',
              'learning_rate': 0.01,
              'subsample': 0.8,
              'subsample_freq': 1,
              'feature_fraction': 0.6,
              # 'max_depth': -1, 
              'num_leaves': 100,
              'min_data_in_leaf': 30,
              'max_bin': 100,
              'n_estimators': 10000,
              'boost_from_average': False,
              'verbose': -1}
$\endgroup$

1 Answer 1

1
$\begingroup$

Generally, if your objective is to maximize predictive performance, you should choose the hyperparameters that maximize your expected predictive performance (eg. by choosing the parameters which lead to maximal performance on holdout data). It's not common to set hyperparameters by hand unless you have some specific goal in mind. If you see significantly better performance with min_data_in_leaf:30 than min_data_in_leaf:100, go with 30, all other things being equal. No need to overthink it.

To your thoughts...

1. I should stick to tuning other parameters that control overfitting and keep min_data_in_leaf fairly low

Never really a bad idea to experiment with more hyperparameters. And in this case, if you know that certain quirks of your dataset will lead to a low value of min_data_in_leaf performing well, it might make extra sense to experiment with other types of regularization. The LightGBM Parameters Tuning page lists around eight other hyperparameters that can reduce overfitting. But again, this should be done in the context of maximizing performance on holdout data. If more regularization leads to a worse predictive model, why are you adding it?

2. Maybe I need to do more feature engineering? (I've tried, unsuccessfully)

That would not solve the problem with min_data_in_leaf. If you only have 40 observations where date='Dec 30' you're only going to have 40 observations where is_dec_30=True.

$\endgroup$
1
  • $\begingroup$ Sincerely appreciate your answer! Yes, I agree that one should not manually choose parameters, I got mine as a result of CV and decided to play around with some of them to better understand my data. Your answer to my question pertaining feature engineering is particularly insightful, I haven’t thought about it that way. Thank you one more time! $\endgroup$
    – User
    May 25, 2022 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.