My situation is:

36,197 observations/ 125 outcomes in training data 26 predictors

A relatively successful prediction model has been built in a similar dataset using just logistic regression; I expect that I have added some informative predictors.

I think I am using reasonable parameters, but I end up with an incredibly overfit model. In my 4-fold stratified cross-validation:

Training F1: 0.957
Test F1:     0.062

Training TPR: 1.000
Test TPR:     0.062


I understand that small eta, small max_depth can help with overfitting -- but my parameter set includes (though does not restrict to) small eta, small max_depth. I used a random grid with 256 combinations of hyperparameters, and I am also using early stopping.

xgb_params_cont <- makeParamSet(
makeIntegerParam("nrounds", lower = 100, upper = 1000),
makeIntegerParam("max_depth", lower = 1, upper = 6),
makeNumericParam("eta", lower = .01, upper = .5),
makeNumericParam("colsample_bytree", lower = .5, upper = 1),
makeNumericParam("gamma", lower = 0, upper = 5),
makeNumericParam("min_child_weight", lower = 1, upper = 5),
makeNumericParam("subsample", lower = 0.6, upper = 0.8),
makeNumericParam("lambda", lower = -0.5, upper = 1, trafo = function(x) 10^x),
makeNumericParam("alpha", lower = -2, upper = 1, trafo = function(x) 10^x),
makeIntegerParam("scale_pos_weight", lower = 100, upper = 3000),
makeIntegerParam("max_delta_step", lower = 0, upper = 10)
)


I have read other posts on overfitting, including Discussion about overfit in xgboost, but I am really confused about where to go from here.

Relatedly, I care both about PPV/precision and recall/sensitivity. This model will be deemed useful if it has a validated PPV of at least 5%, and within that, I'd like to maximize sensitivity (ideally 50% or so) -- I used F1 as my measure of interest. But is that the appropriate way to summarize my wants?

I ask because, in looking at the hyperparameter data, the maximum f1.test.mean = 0.06 and at that value, ppv.test.mean = 0.09 and tpr.test.mean = 0.05 -- which is far lower than is reasonable for my application. There is a hyperparameter combination that results in f1.test.mean = 0.05 and at that value, ppv.test.mean = 0.06 and tpr.test.mean = 0.31. My gut prefers this, though I do not know how to encapsulate that in anything I feed to the model. Using cost-sensitive classification is an option, but including large scale_pos_weight values seemed like an easier way of modifying the objective function.

Thank you all so very much for any suggestions.

The problem you face is that in practice these hyper-parameters are not all independent, so without some insight into the behavior of the model, random search can be quite misleading. I would start with a small grid search centered on sensible starting values for the three most important parameters:

nrounds:  100,200,400,1000
max_depth:    6,10,20
eta:     0.3,0.1,0.05


From this you should be able to get a sense of whether the model benefits from longer rounds, deeper trees, or larger steps.

The only other thing I would say is your regularization values seem large, try leaving them out, then bringing them in at 10^(-5), 10^(-4), 10(-3) scales. Larger than that can make the model brittle.

• Thanks, I will do this. But since posting this, I upped the random grid size to 2048 with similar results -- surely these 'sensible' values must be a subset of a random grid so large? Also, aren't the max_depths you propose quite large? Thank you!
– PBB
Nov 22 '19 at 0:41
• Also, re: regularization values. I accept that mine look large and am happy to implement your suggestions. But higher values make a more conservative model, which is indeed what I want, correct? (Which is not too say that mine might not be too large -- just clarifying for my own understanding.) Thank you.
– PBB
Nov 22 '19 at 1:06
• If you took 3 values for each parameter that would be a grid of 3^11 = 117k points, so 2048 isn't very different from zero :) Now you've got 26 predictors, so a tree which can distinguish all of them would have to be log_2 26 = 4.7 deep. But XGboost is good at pruning, so I would explore all sorts of depths, compared to gbm in R it often seems to work with much deeper trees. You should think of regularization as constraints on the model. It's true that stronger regularization can make more robust models, but if you over constrain the model it can't learn the data properly. Nov 22 '19 at 4:16

Most of your parameter space looks reasonable to me; I'll disagree with @drenerbas about the regularizations being too large. The one that looks most out of place is scale_pos_weight seems too big: since your metrics depend on the cutoff, you probably will want the two classes to be roughly balanced when considering weights, so scale_pos_weight of around 300. But 2/3 of your range is above 1000, rather over-weighting the positive class, which might hurt these metrics, so effectively cutting your number of sampled hyperparameter points by 1/3.

(I would be concerned that your min_child_weights are too small, but limiting to depth 6 mostly obviates that.)

To speak to the process, you should see what hyperparameters are giving you those scores, and see if there's any trend; if all the top scores occur for a certain range of (say) scale_pos_weight, then then shrink that space and run some more. (That can be badly affected by inter-related hyperparameters as @drenerbas says (+1), but it may help.)

Finally, maybe consider a "smarter" optimization algorithm like Bayesian searches.