Is there really any difference between the jackknife and leave one out cross validation? The procedure seems identical am I missing something?
Jackknife often refers to 2 related but different processes, both of which rely on a leave-one-out approach -- leading to this very confusion.
In one context, jackknife can be used to estimate population parameters and their standards errors. For example, to use a jackknife approach to estimate the slope and intercept of a simple regression model one would:
- Estimate the slope and intercept using all available data.
- Leave out 1 observation and estimate the slope and intercept (also known as the "partial estimate" of the coefficients).
- Calculate the difference between the "partial estimate" and the "all data" estimate of the slope and the intercept (also know as the "pseudo value" of the coefficients).
- Repeat steps 2 & 3 for the entire data set.
- Compute the mean of the pseudo values for each coefficient -- these are the jackknife estimates of the slope and intercept
The pseudo values and the jackknife estimates of the coefficients can also be used to determine the standard errors and thus confidence intervals. Typically this approach gives wider confidence intervals for the coefficients because it's a better, more conservative, measure of uncertainty. Also, this approach can be used to get a jackknife estimate of bias for the coefficients too.
In the other context, jackknife is used to evaluate model performance. In this case jackknife = leave-one-out cross validation. Both refer to leaving one observation out of the calibration data set, recalibrating the model, and predicting the observation that was left out. Essentially, each observation is being predicted using its "partial estimates" of the predictors.
Here's a nice little write-up about jackknife I found online: https://www.utdallas.edu/~herve/abdi-Jackknife2010-pretty.pdf