3
$\begingroup$

For a regression problem, I used gradient boosting machines and assessed RMSE. My dataset is comprised of 34 features and 10,000 records. Only 2 predictors were considered 'important' (importance for other predictors happened to be zero): they are both factor features with 300 levels or more, so lots of predictions on new data set gets the same result. When I delete these two features, I get nearly the same RMSE score even if only 3 or 4 predictors highlight some relative influence (again, with zero influence for other predictors).

What could explain this result? Should I be concerned with levels of factor predictors?

Improve this question
$\endgroup$
3
  • $\begingroup$ This question is very unclear. If I understood it better I could clean up the English for you. What is 10000? What kind if regression is GBM (I am not familiar with the acronym)? What does "importance of predictors are just 2" mean? Are you saying that only 2 of the 34 features had coefficients significantly different from 0? also are you saying that if you use models based on just the 32 features with the 2 important ones excluded that you get almost the same RMSE score for each model? Can you explain "however just 3 or 4 predictors have the relative influence"? I don't understand. $\endgroup$
    – Michael R. Chernick
    Sep 16, 2012 at 12:06
  • $\begingroup$ Please answer my questions. If you do I should be able to answer your question and clean up the English to make it easier for others to understand. $\endgroup$
    – Michael R. Chernick
    Sep 16, 2012 at 12:23
  • $\begingroup$ @Chl Your edit cleared up the question and answered all of my questions. Without that help I would have voted to close. Instead I think I can now give a helpful opinion. $\endgroup$
    – Michael R. Chernick
    Sep 16, 2012 at 12:28

1 Answer 1

Reset to default
1
$\begingroup$

It seems that the two important predictors are the only useful ones. Since the other 32 have little or no predictive power it should not be surprising that models excluding these factors have roughly the same (I assume a relatively high) RMSE. Even though some of those models show 3 or 4 of the variables with a relatively high influence I imagine that those 3 or 4 variables are still not good predictors.

Since the two important variables do seem to have explanatory power I think the fact that they both have so many levels may be coincidental rather than suggesting that the high number of levels is the reason for their importance. If you had a small data set the possibility that particular levels of the variable have a chance effect would exist. But since you have 10000 records if the model with just 1 or both of these important variables seems to have predictive power I doubt that the levels are causing this merely by chance.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Sorry for answer you late and my unclear expression.Here is my experiments. First round,the relative influence with levels of factor features A(810 levels 0.6) B(822 levels 0.39) C(a continues variable 0.01) others are zero .Second round,I delete this three predictors,get the result D(continues variable 0.7) E(continues variable 0.2) F(422 levels 0.1). $\endgroup$
    – Chenghao Liu
    Sep 24, 2012 at 16:01
  • $\begingroup$ when i check the the result of gradient boosted tree ,I find most of the tree just have only two important predictors,the depth of tree are 2 or 3. Is that normal? $\endgroup$
    – Chenghao Liu
    Sep 24, 2012 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.