Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For a regression problem, I used gradient boosting machines and assessed RMSE. My dataset is comprised of 34 features and 10,000 records. Only 2 predictors were considered 'important' (importance for other predictors happened to be zero): they are both factor features with 300 levels or more, so lots of predictions on new data set gets the same result. When I delete these two features, I get nearly the same RMSE score even if only 3 or 4 predictors highlight some relative influence (again, with zero influence for other predictors).

What could explain this result? Should I be concerned with levels of factor predictors?

share|improve this question
This question is very unclear. If I understood it better I could clean up the English for you. What is 10000? What kind if regression is GBM (I am not familiar with the acronym)? What does "importance of predictors are just 2" mean? Are you saying that only 2 of the 34 features had coefficients significantly different from 0? also are you saying that if you use models based on just the 32 features with the 2 important ones excluded that you get almost the same RMSE score for each model? Can you explain "however just 3 or 4 predictors have the relative influence"? I don't understand. – Michael Chernick Sep 16 '12 at 12:06
Please answer my questions. If you do I should be able to answer your question and clean up the English to make it easier for others to understand. – Michael Chernick Sep 16 '12 at 12:23
@Chl Your edit cleared up the question and answered all of my questions. Without that help I would have voted to close. Instead I think I can now give a helpful opinion. – Michael Chernick Sep 16 '12 at 12:28

It seems that the two important predictors are the only useful ones. Since the other 32 have little or no predictive power it should not be surprising that models excluding these factors have roughly the same (I assume a relatively high) RMSE. Even though some of those models show 3 or 4 of the variables with a relatively high influence I imagine that those 3 or 4 variables are still not good predictors.

Since the two important variables do seem to have explanatory power I think the fact that they both have so many levels may be coincidental rather than suggesting that the high number of levels is the reason for their importance. If you had a small data set the possibility that particular levels of the variable have a chance effect would exist. But since you have 10000 records if the model with just 1 or both of these important variables seems to have predictive power I doubt that the levels are causing this merely by chance.

share|improve this answer
Sorry for answer you late and my unclear expression.Here is my experiments. First round,the relative influence with levels of factor features A(810 levels 0.6) B(822 levels 0.39) C(a continues variable 0.01) others are zero .Second round,I delete this three predictors,get the result D(continues variable 0.7) E(continues variable 0.2) F(422 levels 0.1). – Chenghao Liu Sep 24 '12 at 16:01
when i check the the result of gradient boosted tree ,I find most of the tree just have only two important predictors,the depth of tree are 2 or 3. Is that normal? – Chenghao Liu Sep 24 '12 at 16:08

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.