3
$\begingroup$

I am trying to predict sales of certain product using regression method. I am using XGboost and using MAPE as final metric for comparison between models. I added a new feature to my existing model and this feature came out to be highly important ( within top 3). But there is no significant change in the final metric MAPE at all. What could be going wrong and what are the possible next steps?

I am performing this in R using xgboost library. Also using xgb.importance() function to get variable importance and using Gain column to understand variable importance. I am modeling time series data and it is at weekly level. It has some time varying components & non time varying features (like static product characteristics) as predictor variables.

$\endgroup$
  • $\begingroup$ Can you tell us more about the structure of your data? Is is cross-sectional data, pannel data or a time series? If it is a time series what is the frequency? Is there a Unit root, etc.? $\endgroup$ – Ferdi Feb 21 '17 at 8:21
  • 1
    $\begingroup$ @Ferdi I have edited my questions to answer your questions on structure of data. $\endgroup$ – navinkb Feb 21 '17 at 8:30
1
$\begingroup$

My first thought would be that your new variable is correlated to the variables you were already using. It does not bring new information but rather 'sums up' the information contained in other variables.

Have you analyzed the correlations of your variables ?

$\endgroup$
  • $\begingroup$ I have similar thoughts on it. The variable is highly correlated to existing variables and is not bringing any new information to the model. $\endgroup$ – user2542275 Feb 21 '17 at 10:06
  • $\begingroup$ I have analyzed the correlations between all my variables. The newly added was more of an ordinal variable. Since my data is already sparse matrix i didn't create dummy variables out of this. Not able to get any other explanation. $\endgroup$ – navinkb Feb 23 '17 at 8:37
0
$\begingroup$

Maybe you have not enough training data, so your model overfits your data. This is called high-variance. To validate it, check your model on the training data and also check it on the test data. If the training error is low and test error is high, it suffers from the high-variance problem and you should use more training data to train a better model that can generalize to the test data. You can also use feature selection methods to remove unimportant features. You can also use regularization to control the complexity of the trained model.

$\endgroup$
  • $\begingroup$ I think i have enough data. Generally 3 years of data is enough for this exercise but i have included around 5 years of data which gave slight improvement. I am trying to tune XGBoost parameters to perform regularization. But it is surprising that addition of a important feature has not affected the final metric $\endgroup$ – navinkb Feb 21 '17 at 9:07
0
$\begingroup$

Beside the reason of it maybe highly correlated with other variables.

There is another reason: it decreases the iteration times (the number of trees) to get the minimum MAPE ( you can do cross validation to confirm that), so it's important features!

$\endgroup$
  • $\begingroup$ Can you elaborate on this ? $\endgroup$ – navinkb Mar 29 '17 at 5:52
  • $\begingroup$ Lately, I have done the related work and met this same problem. I add a new feature which is not highly correlated with other variables, and find that it decrease the iteration times to get the best MAPE, so I think this feature have the combined effect with other variables (tree-model can catch this interact effect), so its feature importance is very high. $\endgroup$ – wolfe Mar 29 '17 at 10:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.