I've implemented a comparison between the performance of 80%-forecast intervals is in the forecast package - see 1st part of the code below providing a number of hits This number states, how many times the forcast interval was right for the left-out data entries. Btw regarding variable names: the German "preis" means "price" and "absatz" means "sales", i.e. "preise" means "prices" and "absaetze" is the plural for "sales".
So, I compared the formula-based prediction interval to what I think bootstrapping is - see 2nd part of the code. But the number of hits in the 2nd case by no means resembles the 80% of the first case. The following actions did not help to reproduce the 80% : using less data in the given data frame, using median formulas for bootstrapping instead of the upper/lower computation in the loop, more samples resampling in the resampling.
I cannot imagine the bootstrapping approach performing so bad - what did I do wrong?
#given
# data frame
preis<-c(1:100)
absatz<-(-2*preis)+1000+rnorm(100)
jeansData<-data.frame(absaetze=absatz,preise=preis)
#### implementation ###
#leave-one-out cross-validation for formula, i.e. with the borders given above
###### (1ST PART) ########
numberOfHits<-0
for(i in (1:100)){
preisCandidateToBeChecked<-preis[i]
absatzCandidateToBeChecked<-absatz[i]
absatzWithoutCandidate<-absatz[-i]
preisWithoutCandidate<-preis[-i]
jeansData<-data.frame (absaetze=absatzWithoutCandidate,preise=preisWithoutCandidate)
fit<-lm((absaetze~preise), data=jeansData)
#check, if in interval and count as hit, if value is in interval
if(absatzCandidateToBeChecked <= (forecast(fit, newdata=preisCandidateToBeChecked)$upper[1]) & (absatzCandidateToBeChecked >= (forecast(fit, newdata=preisCandidateToBeChecked)$lower[1])) )
{numberOfHits<-numberOfHits+1}
}
#execute code until here and inspect numberOfHits; the hit rate pretty much resembles the 80% assumed
#then execute the rest
#leave-one-ot cross-validation for bootstrapping (not using the bootstrap function) ###### (2ND PART) ########
numberOfHits<-0
for(i in (1:100)){
preisCandidateToBeChecked<-preis[i]
absatzCandidateToBeChecked<-absatz[i]
absatzWithoutCandidate<-absatz[-i]
preisWithoutCandidate<-preis[-i]
jeansData<-data.frame(absaetze=absatzWithoutCandidate,preise=preisWithoutCandidate)
#ten or hundred or thousand regressions by bootstrapping
allPredictions<-c()
for(j in (1:10)){
fit<-lm((absaetze~preise), data=jeansData[sample(nrow(jeansData),10,replace=TRUE),])
allPredictions<-c(allPredictions,forecast(fit, newdata=preisCandidateToBeChecked)$mean)
}
#build and name bootstrapped forecast interval from regressions
upper<-sort(allPredictions)[9]
lower<-sort(allPredictions)[2]
if((absatzCandidateToBeChecked <= upper) & (absatzCandidateToBeChecked >= lower) )
{numberOfHits<-numberOfHits+1}
} #inspect numberOfHitsAgain - it's around 40%. What is foul here?!