25
$\begingroup$

Is there really any difference between the jackknife and leave one out cross validation? The procedure seems identical am I missing something?

$\endgroup$

2 Answers 2

17
$\begingroup$

In cross-validation you compute a statistic on the left-out sample(s). Most often, you predict the left-out sample(s) by a model built on the kept samples. In jackknifing, you compute a statistic from the kept samples only.

$\endgroup$
4
  • 7
    $\begingroup$ I do not understand how this answer speaks to the LOOCV in the original question. In what sense can one "compute a statistic" on a single left-out observation? $\endgroup$
    – Alexis
    Feb 18, 2019 at 16:27
  • $\begingroup$ @Alexis I am trying to understand your comment. I am confident that Tommy L meant training a model $y(x)$ based on all-but-one subsample $\mathcal{D}_\bar{i} := \mathcal{D}\setminus \{x_i\}$ and evaluating the prediction on the left-out datapoint $x_i$. Were you pointing out that $\hat y(x_i;\mathcal{D}_\bar{i})$, as a statistic, depends on all datapoints and not just $x_i$? $\endgroup$ Mar 17, 2023 at 1:11
  • $\begingroup$ @paperskilltrees LOOCV means that leave one observation out for each observation in turn, calculate the relevant stats on the remaining observations, and average results across $N$. If I have a sample of size $N$, the statistics are not calculated on the single left out observation in each step, but on the $N - 1$ remaining observations; those $N-1$ observations are not, in any sense, left out. $\endgroup$
    – Alexis
    Mar 20, 2023 at 14:23
  • $\begingroup$ @Alexis Thanks, this is precisely what I thought you meant. $\endgroup$ Mar 20, 2023 at 19:34
23
$\begingroup$

Jackknife often refers to 2 related but different processes, both of which rely on a leave-one-out approach -- leading to this very confusion.

In one context, jackknife can be used to estimate population parameters and their standards errors. For example, to use a jackknife approach to estimate the slope and intercept of a simple regression model one would:

  1. Estimate the slope and intercept using all available data.
  2. Leave out 1 observation and estimate the slope and intercept (also known as the "partial estimate" of the coefficients).
  3. Calculate the difference between the "partial estimate" and the "all data" estimate of the slope and the intercept (also know as the "pseudo value" of the coefficients).
  4. Repeat steps 2 & 3 for the entire data set.
  5. Compute the mean of the pseudo values for each coefficient -- these are the jackknife estimates of the slope and intercept

The pseudo values and the jackknife estimates of the coefficients can also be used to determine the standard errors and thus confidence intervals. Typically this approach gives wider confidence intervals for the coefficients because it's a better, more conservative, measure of uncertainty. Also, this approach can be used to get a jackknife estimate of bias for the coefficients too.

In the other context, jackknife is used to evaluate model performance. In this case jackknife = leave-one-out cross validation. Both refer to leaving one observation out of the calibration data set, recalibrating the model, and predicting the observation that was left out. Essentially, each observation is being predicted using its "partial estimates" of the predictors.

Here's a nice little write-up about jackknife I found online: https://www.utdallas.edu/~herve/abdi-Jackknife2010-pretty.pdf

$\endgroup$
2
  • 2
    $\begingroup$ Unless I am mistaken (and I may well be), your first context describes leave-one-out cross validation. $\endgroup$
    – Alexis
    Feb 18, 2019 at 16:31
  • 3
    $\begingroup$ I was just separating the ideas of estimating parameters using LOO versus estimating the value that was left out (as in LOOCV). I see them as two related but slightly different processes, but perhaps both can be referred to as LOOCV? I could also be mistaken. $\endgroup$
    – jcmb
    Feb 26, 2019 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.