My situation is:
36,197 observations/ 125 outcomes in training data 26 predictors
A relatively successful prediction model has been built in a similar dataset using just logistic regression; I expect that I have added some informative predictors.
I think I am using reasonable parameters, but I end up with an incredibly overfit model. In my 4-fold stratified cross-validation:
Training F1: 0.957 Test F1: 0.062 Training TPR: 1.000 Test TPR: 0.062
I understand that small eta, small max_depth can help with overfitting -- but my parameter set includes (though does not restrict to) small eta, small max_depth. I used a random grid with 256 combinations of hyperparameters, and I am also using early stopping.
xgb_params_cont <- makeParamSet( makeIntegerParam("nrounds", lower = 100, upper = 1000), makeIntegerParam("max_depth", lower = 1, upper = 6), makeNumericParam("eta", lower = .01, upper = .5), makeNumericParam("colsample_bytree", lower = .5, upper = 1), makeNumericParam("gamma", lower = 0, upper = 5), makeNumericParam("min_child_weight", lower = 1, upper = 5), makeNumericParam("subsample", lower = 0.6, upper = 0.8), makeNumericParam("lambda", lower = -0.5, upper = 1, trafo = function(x) 10^x), makeNumericParam("alpha", lower = -2, upper = 1, trafo = function(x) 10^x), makeIntegerParam("scale_pos_weight", lower = 100, upper = 3000), makeIntegerParam("max_delta_step", lower = 0, upper = 10) )
I have read other posts on overfitting, including Discussion about overfit in xgboost, but I am really confused about where to go from here.
Relatedly, I care both about PPV/precision and recall/sensitivity. This model will be deemed useful if it has a validated PPV of at least 5%, and within that, I'd like to maximize sensitivity (ideally 50% or so) -- I used F1 as my measure of interest. But is that the appropriate way to summarize my wants?
I ask because, in looking at the hyperparameter data, the maximum
f1.test.mean = 0.06 and at that value,
ppv.test.mean = 0.09 and
tpr.test.mean = 0.05 -- which is far lower than is reasonable for my application. There is a hyperparameter combination that results in
f1.test.mean = 0.05 and at that value,
ppv.test.mean = 0.06 and
tpr.test.mean = 0.31. My gut prefers this, though I do not know how to encapsulate that in anything I feed to the model. Using cost-sensitive classification is an option, but including large
scale_pos_weight values seemed like an easier way of modifying the objective function.
Thank you all so very much for any suggestions.