I am investigating the relationship between temperature fields obtained from numerical weather models and electricity demand. I am applying a PCA-based approach, i.e. I study the linear relationship between main temperature patterns/modes and main demand patterns/modes. Given that I am working on summer yearly demand, I have time-series with few samples (<20) and for this reason I've decided to apply the following bootstrapping procedure:

  1. I create a temperature and elec. demand datasets with the usual sampling with replacement
  2. I create my linear model between the two fields
  3. I create a temperature dataset with the not-selected samples and I project them on the PCA-space I've just computed
  4. I calculate out-of-sample output

I do this for about 5000 times and in the end I obtain a matrix with only the out-of-sample outputs. I calculate the mean on all the out-of-sample predicted demands for each year and I use it to calculate RMSE error. I think this approach could be considered a .632 bootstrap procedure.

I'd like to compute the significance of the obtained results. I was thinking about the possibility to shuffle at each bootstap iteration the temperature dataset in order to see whether I obtain similar results breaking the direct temporal link between demand and temperature.

Given that I don't have a robust statistical background I'd like your opinion about any method to obtain the statistical significance of my bootstrap procedure.

  • $\begingroup$ Do I understand correctly that you resample years? $\endgroup$ Apr 4, 2013 at 16:49
  • $\begingroup$ If you derive your RMSE only from out-of-sample predictions, you have an out-of-bootstrap estimate. Not .632 because you do not mix in training-set residuals. $\endgroup$ Apr 4, 2013 at 17:03
  • $\begingroup$ Yes, I resample years. Thank you for your comment about .623 procedure. $\endgroup$ Apr 5, 2013 at 6:44

1 Answer 1


I think the crucial point about the "statistical significance" of your results is the implicit assumption, that the years are independent samples of one population. But over 20 years, the temperature-dependent use of electricty may change quite a lot. As really you have a time series of 20 summers, you'd need to look into general trends as well as into seasonal effects, and patterns within the summmers.

  • you can shuffle your data at two or three different levels: within the years (which tells you something about the general relation between electricity use and hot summers) or between the years, i.e. use another year's temperatures with the electricity data.

  • You are talking about time series. Do you generate temperature patterns within the years (e.g. heat wave, considering a lag between temperature and electricity)? If so, you can shuffle those features as well.

  • If you derive your RMSE only from out-of-sample predictions, you have an out-of-bootstrap estimate. Not .632 because you do not mix in training-set residuals.
    Neither would .632 bootstrap be recommended if you have a situation where overfitting can occur. There is also the .632+ bootstrap, which tries to estimate the amount of overfitting, and then adjusts the amount of training set error that is mixed into the estimate.
    Personally, I prefer to stay with the completely independent test set, i.e. pure out-of-bootstrap. If I need to measure overfitting, I do that separately, and prefer to report it separately.

  • $\begingroup$ I don't really understand the 2-3 levels of shuffling, 'within the years' means shuffling the months? $\endgroup$ Apr 6, 2013 at 15:55
  • $\begingroup$ Yes, or the days. Whatever is the most fine-grained level you have. $\endgroup$ Apr 6, 2013 at 16:08
  • $\begingroup$ I somehow assume that the months (weeks, days) end up as different columns in your data. Is that correct? $\endgroup$ Apr 6, 2013 at 16:09
  • $\begingroup$ Currently, I use two-months (June and July) average as columns, but I have daily data too. $\endgroup$ Apr 7, 2013 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.