I'm training an XGBoost binary classifier on a difficult problem from structural microbiology. Due to the (expectedly) low performance of the predictor I plotted learning curves for my training and validation data sets:
The graph was generated through XGBoost's evals_result() method inside a 5-fold stratified shuffled CV loop (code below). This plot looks to me like the predictor is overfitting, due to increasing gap between the training and validation curves (similar example). I tried to prevent this by adding regularization (via various values for XGBoost's subsample and colsample_bytree parameters), early stopping (using a several different early_stopping_rounds) and reducing the learning rate (eta). These either did not work at all or left me with a bad/incomplete predictor (as measured by training data performance). As soon as the training metrics improve, the validation metrics get worse.
My Question
Is this actually overfitting? If so, why aren't my countermeasures working? If not (e.g. because the model is actually not complex enough, i.e. is underfitting), why does the predictor seem to make progress on the training data? If there isn't enough information to decide, what else could I look at?
Other things to note
- the data is fairly balanced between true and false cases with around 15k samples overall; possibly that number could be increased
- the training hasn't converged (the training curve is still decreasing) but that can be addressed by increasing the learning rate and/or number of boosting rounds, both of which I tried and succeeded with, at the expense of an even greater disparity to the validation data
- the features come from one-hot encoded amino acid sequences and are thus quite large (dimension around 2500). I managed to cut that down using dimensionality reduction without any change in the overall trends.
- I didn't do a proper three-fold split of training/validation/test data to report the predictor's final performance because I'm still in the early stages of understanding the impact of my model parameters.
- I also tried sklearn's built-in
learning_curve()function based on this example but I found the graphs less enlightening because they're plotted against number of training samples used, as opposed to boosting rounds. - since XGBoost didn't allow me to automatically run cross-validated
evals_result(), I implemented the CV loop myself so there is the possibility of a bug, although I have tried to take care with the implementation:
model = XGBClassifier(use_label_encoder=False)
cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=42)
# "test" means actually validation here
results = {
'train': [],
'test': [],
}
for train_idx, test_idx in cv.split(X, y):
fold_model = clone(model)
X_train = X[train_idx]
y_train = y[train_idx]
X_test = X[test_idx]
y_test = y[test_idx]
evalset = [(X_train, y_train), (X_test, y_test)]
fold_model.fit(X_train, y_train, eval_metric='logloss', eval_set=evalset)
fold_results = fold_model.evals_result()
results['train'].append(np.array(fold_results['validation_0']['logloss']))
results['test'].append(np.array(fold_results['validation_1']['logloss']))
combined_train = np.concatenate(results['train']).reshape((cv.n_splits, -1))
combined_test = np.concatenate(results['test']).reshape((cv.n_splits, -1))
train_scores_mean = np.mean(combined_train, axis=0)
test_scores_mean = np.mean(combined_test, axis=0)
Update: comparison with Random Forest predictor
It was suggested that this problem might be hopeless. Although it is possible that with my current model, I won't adequately solve the question, it isn't clear to me if the data I have now actually supports that argument. At least it doesn't help me to interpret my learning curves.
After Frans Rodenburg's suggestion, I trained an XGBoost Random Forest classifier with default parameters for comparison. The final logloss I get is 0.636 for training and 0.655 for validation, i.e. both rather high but similar.
Additionally, sklearn's learning_curve() function gives me the following graphs for logloss, however plotted against number of training samples.
Update 2: regularization results
Here is one example of my regularization results, by reducing eta to 0.01 from its default of 0.3:
Although the training and validation (labeled "Test score" in the plot) graphs are much closer, neither converges and both provide terrible performance. For comparison, I can easily achieve a logloss of less than 0.1 for the training data, at which point the curve flattens out. The continuation for additional estimators of the above orange graph is flat. This solution is not a real one; its performance is simply too bad. Similarly, I also tried regularizing via the subsample and colsample_bytree parameters (both down to 0.25) but the graphs don't change their trends at all, while providing worse final performance.
