1
$\begingroup$

I am currently facing an issue in my regression model. I have tried the models in the title (xgbTree, xgbDART, and gbm in caret), and they tend to predict a very small range for the output variable.

The y variable in my train data goes from 2 to 6, and my predicted variable tends to go from 3 to 4.5. I have about 500 samples and 4 predictor variables.

I have also tried very big and very small values for the relevant hyper parameters. I have tried for example:

max_depth : (1, 2, 4, 8, 16, 32)

nrounds: (10, 50, 100, 200, 500, 1000, 2000)

eta: (0.001, 0.01, 0.1, 1)

gamma: (0.0001, 0.001, 0.01, 0.1, 1, 10)

And similar for the other hyperparameters (and for gbm, the relevant ones). My point being that I have tried the extremes of few, deep trees and many, shallow trees. In addition, the hyperparameter grid includes various levels of column sampling and subsampling and various different ranges of minimum number of instances in terminal nodes.

I am doing standard train-test splitting with 10-fold CV for each of the models I tried.

Is my problem just unlearnable?

edit: Forgot to mention. I tried the above with the aformentioned models as stand alone. I have also tried a set up with these models along with other, simpler models as the first tier in a stack ensemble, with a random forest as the second and final layer. Results were the same either way, except the other models in the first layer like xgbLinear and quantile random forest had an easier time covering a bigger range. The overall ensemble still had the small prediction width issue that xgbTree, xgbDART and gbm had.

$\endgroup$
  • $\begingroup$ What is the cost function used? What is the underlying distribution? You may want to try xgboost algorithms with gamma response. (One would have to manually set this parameter to xgbTree.) (+1 though fun question) $\endgroup$ – usεr11852 says Reinstate Monic Sep 7 at 2:03
  • $\begingroup$ @usεr11852 i tried optimizing on both RMSE and Rsquared in the cross validation and my boosting machines. you suggest I switch to reg:gamma for xgbTree and xgbDART? assuming gbm doesn't support it $\endgroup$ – beelze-b Sep 7 at 3:17
  • $\begingroup$ I think that you can define distribution = 'gamma' in gbm; also note that mboost can define: family = GammaReg(). $\endgroup$ – usεr11852 says Reinstate Monic Sep 7 at 11:35
0
$\begingroup$

In most situations, predictions have smaller variance than the response.

More precise: The insample R squared is defined as the variance of the predictions divided by the variance of the response. If the R-squared is smaller than 1, then also your predictions will have less variability than the response. In the worst case, the model always predicts the same value. In this case, the variance of the predictions is 0.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.