I'm looking for a regression model which would satify these requirements:
- My target variable follows the exponential distribution, so to my understanding I should use gamma loss function. I have tried this feature from XGBoost, LightGBM, pyGAM, and StatsModels GLM and they all seem to be working well.
- The model should be able to at least some extrapolation in respect of features $x_e$. In my training data one feature has only values $x_e$ = 1, 2, 3..., 10, and I'm really interested about the prediction at $x_e$=0. Linear models seem to perform well, but tree based models (XGBoost, LightGBM) are very bad at this.
- Some non-linearity or other types of flexibility to fit complex relationships between the input variables. For example with BayesianRidge I get almost the same performance in training and test sets, which tells me that I should be able to build a better model by adding complexity.
My data set contains around one million samples and I'm using 10 features (both discrete and continuous). Potentially I might get 10-20 new features in the future.
Here are the results I have so far. Since I want the model to perform well on the extrapolation in respect of input x, my test set consists of all the samples with $x_e$=1.
- Linear regression softplus link Training loss: 0.949 Test loss: 0.673
- Bayesin Ridge regression Training loss: 0.931 Test loss: 0.653
- Bayesin Ridge with softplus link Training loss: 0.949 Test loss: 0.673
- Elastic Net Training loss: 0.930 Test loss: 0.660
- Elastic Net with softplus link Training loss: 0.946 Test loss: 0.655
- Elastic Net CV Training loss: 0.931 Test loss: 0.654
- Elastic Net CV with log link Training loss: 0.984 Test loss: 0.688
- Elastic Net CV with softplus link Training loss: 0.947 Test loss: 0.664
- GLM with log link and gamma loss Training loss: 0.927 Test loss: 0.657
- GAM with log link gamma loss Training loss: 0.927 Test loss: 0.657
- LGBMRegressor (max_depth 3) Training loss: 0.899 Test loss: 0.678
All models seem to have much better loss at the test set than the training set. Normally this would indicate over fitting, but I believe that the reason is that the values at $x_e$=1 are lower than those at other values of $x_e$.
Initially I had a completely randomized train-test split. Based on those results LGBM was model for interpolation, but now it is clear that the other models outperform it in extrapolation. Based on those results, none of the current models were overfit. I think that the model should have only linear or other very simple dependency to $x_e$, but there is probably potential to increase the model complexity in respect of the other features.