Overfitting in extreme gradient boosting My situation is:
36,197 observations/ 125 outcomes in training data
26 predictors
A relatively successful prediction model has been built in a similar dataset using just logistic regression; I expect that I have added some informative predictors.
I think I am using reasonable parameters, but I end up with an incredibly overfit model. In my 4-fold stratified cross-validation:
Training F1: 0.957
Test F1:     0.062

Training TPR: 1.000
Test TPR:     0.062

I understand that small eta, small max_depth can help with overfitting -- but my parameter set includes (though does not restrict to) small eta, small max_depth. I used a random grid with 256 combinations of hyperparameters, and I am also using early stopping.
xgb_params_cont <- makeParamSet(
  makeIntegerParam("nrounds", lower = 100, upper = 1000),
  makeIntegerParam("max_depth", lower = 1, upper = 6),
  makeNumericParam("eta", lower = .01, upper = .5),
  makeNumericParam("colsample_bytree", lower = .5, upper = 1),
  makeNumericParam("gamma", lower = 0, upper = 5),
  makeNumericParam("min_child_weight", lower = 1, upper = 5),
  makeNumericParam("subsample", lower = 0.6, upper = 0.8),
  makeNumericParam("lambda", lower = -0.5, upper = 1, trafo = function(x) 10^x),
  makeNumericParam("alpha", lower = -2, upper = 1, trafo = function(x) 10^x),
  makeIntegerParam("scale_pos_weight", lower = 100, upper = 3000),
  makeIntegerParam("max_delta_step", lower = 0, upper = 10)
)

I have read other posts on overfitting, including  Discussion about overfit in xgboost, but I am really confused about where to go from here.
Relatedly, I care both about PPV/precision and recall/sensitivity. This model will be deemed useful if it has a validated PPV of at least 5%, and within that, I'd like to maximize sensitivity (ideally 50% or so) -- I used F1 as my measure of interest. But is that the appropriate way to summarize my wants?
I ask because, in looking at the hyperparameter data, the maximum f1.test.mean = 0.06 and at that value, ppv.test.mean = 0.09 and tpr.test.mean = 0.05 -- which is far lower than is reasonable for my application. There is a hyperparameter combination that results in f1.test.mean = 0.05 and at that value, ppv.test.mean = 0.06 and tpr.test.mean = 0.31. My gut prefers this, though I do not know how to encapsulate that in anything I feed to the model. Using cost-sensitive classification is an option, but including large scale_pos_weight values seemed like an easier way of modifying the objective function.
Thank you all so very much for any suggestions.
 A: The problem you face is that in practice these hyper-parameters are not all independent, so without some insight into the behavior of the model, random search can be quite misleading. I would start with a small grid search centered on sensible starting values for the three most important parameters:
nrounds:  100,200,400,1000
max_depth:    6,10,20
eta:     0.3,0.1,0.05

From this you should be able to get a sense of whether the model benefits from longer rounds, deeper trees, or larger steps.
The only other thing I would say is your regularization values seem large, try leaving them out, then bringing them in at 10^(-5), 10^(-4), 10(-3) scales. Larger than that can make the model brittle.
A: Most of your parameter space looks reasonable to me; I'll disagree with @drenerbas about the regularizations being too large.  The one that looks most out of place is scale_pos_weight seems too big: since your metrics depend on the cutoff, you probably will want the two classes to be roughly balanced when considering weights, so scale_pos_weight of around 300.  But 2/3 of your range is above 1000, rather over-weighting the positive class, which might hurt these metrics, so effectively cutting your number of sampled hyperparameter points by 1/3.
(I would be concerned that your min_child_weights are too small, but limiting to depth 6 mostly obviates that.)
To speak to the process, you should see what hyperparameters are giving you those scores, and see if there's any trend; if all the top scores occur for a certain range of (say) scale_pos_weight, then then shrink that space and run some more.  (That can be badly affected by inter-related hyperparameters as @drenerbas says (+1), but it may help.)  
Finally, maybe consider a "smarter" optimization algorithm like Bayesian searches.
