I have recently been doing work with predictive models for a continuous response. I am doing a comparison between Elastic Net (
glmnet) package in R and XGBoost (
xgboost) package in R. Originally, I built the model using Elastic Net for its ability to perform feature selection and also for its ability to shrink the coefficeints of correlated variables.
I am exploring XGBoost because of its predictive capabilities, the summary of feature importance it provides, its ability to capture non-linear interactions and also because I believe that it might be more robust in the presence of outliers.
My questions are:
Is XGBoost or gradient boosted trees in general better at finding non-linear interactions than a generalized linear model?
Is my assumption about XGBoost or gradient boosted trees in general being robust to outliers a fair assumption?
Here is my model set up and finding:
For model validation I have a training and testing set. I $log$ transform the response variables before model fitting. I make predictions on the testing set and then exponentiate the results to return to the original scale. I make predicted vs. observed plots for each model.
The exponentiated predicted values have some outliers but the fit in general is good.
With the elastic net model when I convert back to original scale there is an extreme predicted value. I am interpreting this as that the GLM NET has a few cases that it is not quite sure how to predict (outliers).
I would love to hear opinions! Thank you in advance for any help or comments!