I'm using gradient boosting regression model (GBRT).

To evaluate this model, I use 10-fold cross validation, in each of which I set same param and compute the coefficient of determination as a measure of fitting.

However, I find that there exists a huge difference in coefficient of determination obtained from each fold, e.g., the coefficient of determination from fold_1 to fold_10 is:

[ 0.95310245  0.89725342  0.886711    **0.97063794**  0.84182142  0.80870443
  0.70535911  0.8888032   **0.42510782**  0.70421155]

Although the mean is 0.81 and std is 0.31, there is a fold in which the coefficient of determination is 0.4, while another fold is 0.97.

The only difference btw each fold is just the training & test data set, why does there exist such huge difference? Is such difference indicating that the performance of my model is not good?

  • $\begingroup$ It certainly indicates some sort of problem, but what the problem is is impossible to determine from the limited information you provide. You should look at each model, but particularly the one with the very low value, and see what is going on. $\endgroup$ – Peter Flom - Reinstate Monica Sep 12 '14 at 11:50
  • $\begingroup$ @PeterFlom Thanks, but I set same parameter for each fold, thus the only difference among folds is the data set. Any suggestion to do next in order to find out the problem? $\endgroup$ – ice_lin Sep 12 '14 at 12:11
  • $\begingroup$ Provide the sample size, distribution of $Y$, and total number of parameters being fitted or entertained. $\endgroup$ – Frank Harrell Sep 12 '14 at 13:12

Given your response, something is clearly odd in your data, at least for your model. You probably have outliers, influential points, leverage points etc. Have you looked for these?

If you are using SAS, PROC ROBUSTREG offers some nice diagnostics. Doubtless there are also some in R, but I don't know them.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.