I have the value of a dependent variable, Y, for every day from 1/1/2000 to 12/31/2016. The value is partially dependent on the day of the year/week/month, the month, and the year; as well as on other variables. I'm training my model using 2000-2015 and testing using 2016. If I have a model that predicts Y based on the day and factors X[1]...X[N], how do I determine whether any added accuracy from new factor X[N+1] is statistically significant. If it weren't for the date dependency, I would use bootstrap or other random sampling techniques across multiple runs. It doesn't seem like removing random dates or days of the year is a good approach.

  • $\begingroup$ If you're using regression of some kind, look into what's variously called a sequential or multi-stage or hierarchical use of regression. That will show, beyond what the other factors can do, how much the new factor can do to predict Y. $\endgroup$ – rolando2 Jan 19 '17 at 20:02
  • $\begingroup$ Thanks, Rolando2. I will look into that, but I'm using neural networks and XGBoost, so I really need something that just looks at the result of the loss function for each run. $\endgroup$ – Jim Cutler Jan 19 '17 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.