0
$\begingroup$

I have a regression model using XGBoost that I was getting great MAE and MAPE results on my test dataset.

mape: 2.515660669106389
mae: 90591.77886478149

Thinking that it was too good to be true, I ran 10-fold cross validation on the train dataset, and got the following results and distribution in the results. Results are plotted by binning them into 10 bins.

from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score

xg = XGBRegressor(learning_rate=0.5, n_estimators=50, max_depth=4, random_state=4)
kfold = KFold(n_splits=10, random_state=7)
results = cross_val_score(xg, X_train, Y_train, cv=kfold, scoring='neg_mean_absolute_error') 

results_y = scaler_y.inverse_transform(np.abs(results.reshape(-1,1)))
print(results_y)
plt.hist(results_y, bins=20)
plt.ylabel('MAE')
plt.show()

Results (MAE):

[[1737985.90765678]
 [ 466277.11674066]
 [  47184.70876369]
 [ 129014.99538841]
 [  23133.30322564]
 [  44112.92209214]
 [  69724.235821  ]
 [ 119278.83633742]
 [  39059.981985  ]
 [   8856.48620648]]

enter image description here

So my questions are:

1) Have I over-trained on my test dataset, for some reason?

2) Is the distribution of the cross validated results reasonable? If it is not reasonable, what should I be seeing?

3) If I have over-trained for some reason, what are the ways to mitigate this? What could be some of the reasons? Specifically with regards to XGBoost.

Thank you.

$\endgroup$
  • 1
    $\begingroup$ Have you checked whether 1 outlier could be skewing your results? This is generally less of a problem when using MAE than least squares, but it's worth a quick check, to see whether there is one data point whose value is order of magnitude larger than the others. Or you could calculate MAE for each test point and take the mean of the smallest 95% rather than the mean of all (would require you to slightly customise your cross-validation function) $\endgroup$ – gazza89 Sep 26 '18 at 9:48
  • $\begingroup$ @gazza89 I have visually seen that there's no outliers, it is a somewhat smooth upwards curve. (there aren't that many data points) $\endgroup$ – lppier Sep 28 '18 at 0:55
  • $\begingroup$ @gazza89 For your second part, do you mean to exclude some of the larger errors? As in, keeping only the more accurate 95%? $\endgroup$ – lppier Sep 28 '18 at 1:06
  • $\begingroup$ Yes, just to see whether a small number of data points are disproportionately skewing your results. This shouldn't be the case when using MAE (it often is when using least squares), but it's always a worthwhile and simple check $\endgroup$ – gazza89 Sep 28 '18 at 9:04
  • 1
    $\begingroup$ How large is your dataset? $\endgroup$ – jbowman Sep 28 '18 at 18:36
3
+25
$\begingroup$

1) One does not overtrain on the test set - one overtrains on the training set, and that may cause the regressor to generalize poorly, which results in large errors for the test set. This is called overfitting But you claim that the errors for the test set were low, so no problem there.

So no, no overfitting .

2) The distribution of the CV results are somewhat strange - as the size of the folds increase, one should expect that the distribution of the CV results approaches a Gaussian, centered around the "correct" MAE, which would be the MAE you obtain for the test set. But your folds are not large. You said that the data set has 200 data, but how many of those are in the test set and how many in the training set - the folds are done only on the training set from what I saw in the code. So the folds are at most 20 data, and that could explain the skewness of the distribution of CV results. It seems that your regressor is making some large mistakes on a few data points, and that was the inspiration for some of the comments asking you about outliers - the few data points for which the regressor is making very wrong predictions.

3) XGBoost and other boosting algorithms tend to overfit the training set (but we discussed that there is no overfit because you think that the results in the test set are good). Boosting learns a regressor on the data, than a new regressosn of the errors of the first one, than a new one on the errors of the previous two, and so on. This tend to create an ensemble regressor that fits very well the training set which may cause poor generalization, which may explain large errors for the test set. In your case you are generating 50 of those regressors (which seems too much given that there is only 200 data) and each regressor will probably no make that many mistakes (given that each regressor is a regressor tree with at most 4 levels deep). The usual way to reduce overfitting on boosting is to a) reduce the number of estimators, and/or b) reduce the maximum depth of the trees.

I do not have enough experience on boosting to advice you on whether your values are too high, I would suspect so.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.