Out of sample bootstrapping and significance

Question

I am investigating the relationship between temperature fields obtained from numerical weather models and electricity demand. I am applying a PCA-based approach, i.e. I study the linear relationship between main temperature patterns/modes and main demand patterns/modes. Given that I am working on summer yearly demand, I have time-series with few samples (<20) and for this reason I've decided to apply the following bootstrapping procedure:

1. I create a temperature and elec. demand datasets with the usual sampling with replacement
2. I create my linear model between the two fields
3. I create a temperature dataset with the not-selected samples and I project them on the PCA-space I've just computed
4. I calculate out-of-sample output

I do this for about 5000 times and in the end I obtain a matrix with only the out-of-sample outputs. I calculate the mean on all the out-of-sample predicted demands for each year and I use it to calculate RMSE error. I think this approach could be considered a .632 bootstrap procedure.

I'd like to compute the significance of the obtained results. I was thinking about the possibility to shuffle at each bootstap iteration the temperature dataset in order to see whether I obtain similar results breaking the direct temporal link between demand and temperature.

Given that I don't have a robust statistical background I'd like your opinion about any method to obtain the statistical significance of my bootstrap procedure.

Answer 1

I think the crucial point about the "statistical significance" of your results is the implicit assumption, that the years are independent samples of one population. But over 20 years, the temperature-dependent use of electricty may change quite a lot. As really you have a time series of 20 summers, you'd need to look into general trends as well as into seasonal effects, and patterns within the summmers.

• you can shuffle your data at two or three different levels: within the years (which tells you something about the general relation between electricity use and hot summers) or between the years, i.e. use another year's temperatures with the electricity data.

• You are talking about time series. Do you generate temperature patterns within the years (e.g. heat wave, considering a lag between temperature and electricity)? If so, you can shuffle those features as well.

• If you derive your RMSE only from out-of-sample predictions, you have an out-of-bootstrap estimate. Not .632 because you do not mix in training-set residuals.
  Neither would .632 bootstrap be recommended if you have a situation where overfitting can occur. There is also the .632+ bootstrap, which tries to estimate the amount of overfitting, and then adjusts the amount of training set error that is mixed into the estimate.
  Personally, I prefer to stay with the completely independent test set, i.e. pure out-of-bootstrap. If I need to measure overfitting, I do that separately, and prefer to report it separately.