For a regression problem, I used gradient boosting machines and assessed RMSE. My dataset is comprised of 34 features and 10,000 records. Only 2 predictors were considered 'important' (importance for other predictors happened to be zero): they are both factor features with 300 levels or more, so lots of predictions on new data set gets the same result. When I delete these two features, I get nearly the same RMSE score even if only 3 or 4 predictors highlight some relative influence (again, with zero influence for other predictors).
What could explain this result? Should I be concerned with levels of factor predictors?