Alright, hopefully third time is the charm. I'm basically trying to build a predictive model with R and gbm. For various reasons, I can't explicitly state exactly what I'm doing.
Basically I have a response variable that averages around zero. It has high volatility, is right skewed, and has very high kurtosis. I have a bunch of predictors I think contain information about the response. Some of them are numerical, others factors. I have partitioned available data into three sets 1, 2, 3. What I've done is trained gbm on set 1 with parameters as follows: interaction depth of 10, shrinkage of .001, 3 fold cross validation, about 200000 iterations (maybe overkill), out of bag estimation 50%, and training % at 80%. I've then tested how well the model predicts the responses in set 2 and it's significantly better than chance. The predictors are distributed similarly to the response (similar averages), but the responses have a higher stdev and kurtosis. I then proceed to update the model by training on set 1 + set 2 with the same parameters. I then test on set 3. The predictors now are extremely different from the actual responses (average of 25 for the predictor versus close to 0 for responses). I'm not sure what's causing this. I haven't changed anything so my initial thought was the data in set 2, where the range of the response variable is larger than set 1. How do I narrow down what might be causing this discrepancy with the predictions for set 3?
I realize this is somewhat open-ended, so please let me know what other information would be helpful in answering this question. I'm not sure what exactly you guys might find useful.
answers to comments the data is a time series so just took three time period of about 2 years, 1 year, and 3 months.