Alright, hopefully third time is the charm. I'm basically trying to build a predictive model with R and gbm. For various reasons, I can't explicitly state exactly what I'm doing.

Basically I have a response variable that averages around zero. It has high volatility, is right skewed, and has very high kurtosis. I have a bunch of predictors I think contain information about the response. Some of them are numerical, others factors. I have partitioned available data into three sets 1, 2, 3. What I've done is trained gbm on set 1 with parameters as follows: interaction depth of 10, shrinkage of .001, 3 fold cross validation, about 200000 iterations (maybe overkill), out of bag estimation 50%, and training % at 80%. I've then tested how well the model predicts the responses in set 2 and it's significantly better than chance. The predictors are distributed similarly to the response (similar averages), but the responses have a higher stdev and kurtosis. I then proceed to update the model by training on set 1 + set 2 with the same parameters. I then test on set 3. The predictors now are extremely different from the actual responses (average of 25 for the predictor versus close to 0 for responses). I'm not sure what's causing this. I haven't changed anything so my initial thought was the data in set 2, where the range of the response variable is larger than set 1. How do I narrow down what might be causing this discrepancy with the predictions for set 3?

I realize this is somewhat open-ended, so please let me know what other information would be helpful in answering this question. I'm not sure what exactly you guys might find useful.

answers to comments the data is a time series so just took three time period of about 2 years, 1 year, and 3 months.


migrated from stackoverflow.com Sep 22 '11 at 13:43

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ It's hard to make any reasoning without knowing actual source and/or nature of your data. $\endgroup$ – ffriend Sep 21 '11 at 22:32
  • $\begingroup$ I hope you can understand why this is an impossible question in its current form. It basically says "I made a mistake. Tell me what I did wrong." $\endgroup$ – DWin Sep 21 '11 at 22:59
  • $\begingroup$ I'm trying to model a response variable that averages about 0. That's the most informative part of your question. Could you develop it? $\endgroup$ – Fezvez Sep 21 '11 at 23:53
  • $\begingroup$ You cannot give details -- ok, but your question is a total mess. Please, try to clearly formulate your problem instead of adding redundant details like predictor distributions or your speculations. $\endgroup$ – user88 Sep 22 '11 at 16:42
  • $\begingroup$ Exactly how did you partition the data into those three sets? $\endgroup$ – whuber Sep 23 '11 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy