Skip to main content
Cross Validated

Return to Answer

edited body
Source Link
max0r
max0r
  • 111
  • 4

This sounds like a great application for the $k$-fold cross-validated paired $t$ test (Dietterich, 19971998). However, you need the accuracies per fold of both your model and the model from the other paper. Having only the mean accuracies over all folds is not sufficient (see alternative below).

The $k$-fold cross-validated paired $t$-test test

We have two classification machine learning models $A$ and $B$, that we wish to compare. Under the null hypothesis, both models should have the same test set accuracies $p_A = p_B$. Dietterich (19971998) refers to error rates, which is simply $1-$accuracy. The accuracy $p$ is the probability that a randomly drawn sample from the test set will be classified correctly.

To compare both models, we are interested in the difference of the accuracies $p = p_A - p_B$. First, we perform $k$-fold cross-validation by splitting the dataset into $k$ disjoint sets of equal size. We then perform $ k $ experiments, where each set is used as test set and the remaining sets are used for training. Assuming that the $ k $ differences $ p^{(i)} = p_A^{(i)}-p_B^{(i)} $ are drawn independently from a normal distribution, we can perform Student's $ t $-test test by computing the statistic

$$ t = \frac{\bar{p}\sqrt{k}}{\sqrt{\frac{\sum_{i=1}^{k} (p^{(i)}-\bar{p})^2}{k-1}}} ~ , $$ where $ \bar{p} = \frac{1}{k} \sum_{i=1}^{k} p^{(i)} $. For $ k=10 $ experiments (10-fold cross-validation), the test statistic has a $t$ distribution with $ k-1=9 $ degrees of freedom. The null hypothesis can be rejected if $ |t| > t_{9,0.975} = 2.262 $ (for a two-sided test with the probability of incorrectly rejecting the null hypothesis of $0.05$).

But what if I only have the mean accuracy over all folds?

In this case, you can still perform the following test. Let $p_A$ and $p_B$ be the mean of the accuracies over $k$ folds, which are assumed to be normally distributed. If the two accuracies are independent, the difference $ p_A - p_B $ is also normally distributed. Assuming that the null hypothesis is true, we can compute the following statistic

$$ z = \frac{p_A - p_B}{\sqrt{2p(1-p)/n}} ~ , $$ where $ p=(p_A + p_B)/2 $ and number of test samples $n$. This test statistic has approximately a standard normal distribution and we can reject the null hypothesis if $ |z| > Z_{0.975} = 1.96 $.

However, this test has several problems as outlined by Dietterich (19971998), e.g., $p_A$ and $p_B$ are not independent. Ideally, one should perform the proposed 5x2cv paired $t$-test test, if retraining and evaluation of both models being compared is possible.

Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7), 1895-1923. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.3325&rep=rep1&type=pdf

This sounds like a great application for the $k$-fold cross-validated paired $t$ test (Dietterich, 1997). However, you need the accuracies per fold of both your model and the model from the other paper. Having only the mean accuracies over all folds is not sufficient (see alternative below).

The $k$-fold cross-validated paired $t$-test

We have two classification machine learning models $A$ and $B$, that we wish to compare. Under the null hypothesis, both models should have the same test set accuracies $p_A = p_B$. Dietterich (1997) refers to error rates, which is simply $1-$accuracy. The accuracy $p$ is the probability that a randomly drawn sample from the test set will be classified correctly.

To compare both models, we are interested in the difference of the accuracies $p = p_A - p_B$. First, we perform $k$-fold cross-validation by splitting the dataset into $k$ disjoint sets of equal size. We then perform $ k $ experiments, where each set is used as test set and the remaining sets are used for training. Assuming that the $ k $ differences $ p^{(i)} = p_A^{(i)}-p_B^{(i)} $ are drawn independently from a normal distribution, we can perform Student's $ t $-test by computing the statistic

$$ t = \frac{\bar{p}\sqrt{k}}{\sqrt{\frac{\sum_{i=1}^{k} (p^{(i)}-\bar{p})^2}{k-1}}} ~ , $$ where $ \bar{p} = \frac{1}{k} \sum_{i=1}^{k} p^{(i)} $. For $ k=10 $ experiments (10-fold cross-validation), the test statistic has a $t$ distribution with $ k-1=9 $ degrees of freedom. The null hypothesis can be rejected if $ |t| > t_{9,0.975} = 2.262 $ (for a two-sided test with the probability of incorrectly rejecting the null hypothesis of $0.05$).

But what if I only have the mean accuracy over all folds?

In this case, you can still perform the following test. Let $p_A$ and $p_B$ be the mean of the accuracies over $k$ folds, which are assumed to be normally distributed. If the two accuracies are independent, the difference $ p_A - p_B $ is also normally distributed. Assuming that the null hypothesis is true, we can compute the following statistic

$$ z = \frac{p_A - p_B}{\sqrt{2p(1-p)/n}} ~ , $$ where $ p=(p_A + p_B)/2 $ and number of test samples $n$. This test statistic has approximately a standard normal distribution and we can reject the null hypothesis if $ |z| > Z_{0.975} = 1.96 $.

However, this test has several problems as outlined by Dietterich (1997), e.g., $p_A$ and $p_B$ are not independent. Ideally, one should perform the proposed 5x2cv paired $t$-test, if retraining and evaluation of both models being compared is possible.

Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7), 1895-1923. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.3325&rep=rep1&type=pdf

This sounds like a great application for the $k$-fold cross-validated paired $t$ test (Dietterich, 1998). However, you need the accuracies per fold of both your model and the model from the other paper. Having only the mean accuracies over all folds is not sufficient (see alternative below).

The $k$-fold cross-validated paired $t$ test

We have two classification machine learning models $A$ and $B$, that we wish to compare. Under the null hypothesis, both models should have the same test set accuracies $p_A = p_B$. Dietterich (1998) refers to error rates, which is simply $1-$accuracy. The accuracy $p$ is the probability that a randomly drawn sample from the test set will be classified correctly.

To compare both models, we are interested in the difference of the accuracies $p = p_A - p_B$. First, we perform $k$-fold cross-validation by splitting the dataset into $k$ disjoint sets of equal size. We then perform $ k $ experiments, where each set is used as test set and the remaining sets are used for training. Assuming that the $ k $ differences $ p^{(i)} = p_A^{(i)}-p_B^{(i)} $ are drawn independently from a normal distribution, we can perform Student's $ t $ test by computing the statistic

$$ t = \frac{\bar{p}\sqrt{k}}{\sqrt{\frac{\sum_{i=1}^{k} (p^{(i)}-\bar{p})^2}{k-1}}} ~ , $$ where $ \bar{p} = \frac{1}{k} \sum_{i=1}^{k} p^{(i)} $. For $ k=10 $ experiments (10-fold cross-validation), the test statistic has a $t$ distribution with $ k-1=9 $ degrees of freedom. The null hypothesis can be rejected if $ |t| > t_{9,0.975} = 2.262 $ (for a two-sided test with the probability of incorrectly rejecting the null hypothesis of $0.05$).

But what if I only have the mean accuracy over all folds?

In this case, you can still perform the following test. Let $p_A$ and $p_B$ be the mean of the accuracies over $k$ folds, which are assumed to be normally distributed. If the two accuracies are independent, the difference $ p_A - p_B $ is also normally distributed. Assuming that the null hypothesis is true, we can compute the following statistic

$$ z = \frac{p_A - p_B}{\sqrt{2p(1-p)/n}} ~ , $$ where $ p=(p_A + p_B)/2 $ and number of test samples $n$. This test statistic has approximately a standard normal distribution and we can reject the null hypothesis if $ |z| > Z_{0.975} = 1.96 $.

However, this test has several problems as outlined by Dietterich (1998), e.g., $p_A$ and $p_B$ are not independent. Ideally, one should perform the proposed 5x2cv paired $t$ test, if retraining and evaluation of both models being compared is possible.

Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7), 1895-1923. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.3325&rep=rep1&type=pdf

added 5 characters in body
Source Link
max0r
max0r
  • 111
  • 4

This sounds like a great application for the $k$-fold cross-validated paired $t$ test (Dietterich, 1997). However, you need the accuracies per fold of both your model and the model from the other paper. Having only the mean accuracies over all folds is not sufficient (see alternative below).

The $k$-fold cross-validated paired $t$-test

We have two classification machine learning models $A$ and $B$, that we wish to compare. Under the null hypothesis, both models should have the same test set accuracies $p_A = p_B$. Dietterich (1997) refers to error rates, which is simply $1-$accuracy. The accuracy $p$ is the probability that a randomly drawn sample from the test set will be classified correctly.

To compare both models, we are interested in the difference of the accuracies $p = p_A - p_B$. First, we perform $k$-fold cross-validation by splitting the dataset into $k$ disjoint sets of equal size. We then perform $ k $ experiments, where each set is used as test set and the remaining sets are used for training. Assuming that the $ k $ differences $ p^{(i)} = p_A^{(i)}-p_B^{(i)} $ are drawn independently from a normal distribution, we can perform Student's $ t $ test-test by computing the statistic

$$ t = \frac{\bar{p}\sqrt{k}}{\sqrt{\frac{\sum_{i=1}^{k} (p^{(i)}-\bar{p})^2}{k-1}}} ~ , $$ where $ \bar{p} = \frac{1}{k} \sum_{i=1}^{k} p^{(i)} $. For $ k $$ k=10 $ experiments (10-fold cross-validation), the test statistic has a $t$ distribution with $ k-1 $$ k-1=9 $ degrees of freedom. The null hypothesis can be rejected if $ |t| > t_{9,0.975} = 2.262 $ (for a two-sided test with the probability of incorrectly rejecting the null hypothesis of $0.05$).

But what if I only have the mean accuracy over all folds?

In this case, you can still perform the following test. Let $p_A$ and $p_B$ be the mean of the accuracies over $k$ folds, which are assumed to be normally distributed. If the two accuracies are independent, the difference $ p_A - p_B $ is also normally distributed. Assuming that the null hypothesis is true, we can compute the following statistic

$$ z = \frac{p_A - p_B}{\sqrt{2p(1-p)/n}} ~ , $$ where $ p=(p_A + p_B)/2 $ and number of test samples $n$. This test statistic has approximately a standard normal distribution and we can reject the null hypothesis if $ |z| > Z_{0.975} = 1.96 $.

However, this test has several problems as outlined by Dietterich (1997), e.g., $p_A$ and $p_B$ are not independent. Ideally, one should perform the proposed 5x2cv paired $t$ test-test, if retraining and evaluation of both models being compared is possible.

Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7), 1895-1923. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.3325&rep=rep1&type=pdf

This sounds like a great application for the $k$-fold cross-validated paired $t$ test (Dietterich, 1997). However, you need the accuracies per fold of both your model and the model from the other paper. Having only the mean accuracies over all folds is not sufficient (see alternative below).

The $k$-fold cross-validated paired $t$-test

We have two classification machine learning models $A$ and $B$, that we wish to compare. Under the null hypothesis, both models should have the same test set accuracies $p_A = p_B$. Dietterich (1997) refers to error rates, which is simply $1-$accuracy. The accuracy $p$ is the probability that a randomly drawn sample from the test set will be classified correctly.

To compare both models, we are interested in the difference of the accuracies $p = p_A - p_B$. First, we perform $k$-fold cross-validation by splitting the dataset into $k$ disjoint sets of equal size. We then perform $ k $ experiments, where each set is used as test set and the remaining sets are used for training. Assuming that the $ k $ differences $ p^{(i)} = p_A^{(i)}-p_B^{(i)} $ are drawn independently from a normal distribution, we can perform Student's $ t $ test by computing the statistic

$$ t = \frac{\bar{p}\sqrt{k}}{\sqrt{\frac{\sum_{i=1}^{k} (p^{(i)}-\bar{p})^2}{k-1}}} ~ , $$ where $ \bar{p} = \frac{1}{k} \sum_{i=1}^{k} p^{(i)} $. For $ k $ experiments, the test statistic has a $t$ distribution with $ k-1 $ degrees of freedom. The null hypothesis can be rejected if $ |t| > t_{9,0.975} = 2.262 $ (for a two-sided test with the probability of incorrectly rejecting the null hypothesis of $0.05$).

But what if I only have the mean accuracy over all folds?

In this case, you can perform the following test. Let $p_A$ and $p_B$ be the mean of the accuracies over $k$ folds, which are assumed to be normally distributed. If the two accuracies are independent, the difference $ p_A - p_B $ is also normally distributed. Assuming that the null hypothesis is true, we can compute the following statistic

$$ z = \frac{p_A - p_B}{\sqrt{2p(1-p)/n}} ~ , $$ where $ p=(p_A + p_B)/2 $ and number of test samples $n$. This test statistic has approximately a standard normal distribution and we can reject the null hypothesis if $ |z| > Z_{0.975} = 1.96 $.

However, this test has several problems as outlined by Dietterich (1997), e.g., $p_A$ and $p_B$ are not independent. Ideally, one should perform the proposed 5x2cv paired $t$ test, if retraining and evaluation of both models being compared is possible.

Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7), 1895-1923. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.3325&rep=rep1&type=pdf

This sounds like a great application for the $k$-fold cross-validated paired $t$ test (Dietterich, 1997). However, you need the accuracies per fold of both your model and the model from the other paper. Having only the mean accuracies over all folds is not sufficient (see alternative below).

The $k$-fold cross-validated paired $t$-test

We have two classification machine learning models $A$ and $B$, that we wish to compare. Under the null hypothesis, both models should have the same test set accuracies $p_A = p_B$. Dietterich (1997) refers to error rates, which is simply $1-$accuracy. The accuracy $p$ is the probability that a randomly drawn sample from the test set will be classified correctly.

To compare both models, we are interested in the difference of the accuracies $p = p_A - p_B$. First, we perform $k$-fold cross-validation by splitting the dataset into $k$ disjoint sets of equal size. We then perform $ k $ experiments, where each set is used as test set and the remaining sets are used for training. Assuming that the $ k $ differences $ p^{(i)} = p_A^{(i)}-p_B^{(i)} $ are drawn independently from a normal distribution, we can perform Student's $ t $-test by computing the statistic

$$ t = \frac{\bar{p}\sqrt{k}}{\sqrt{\frac{\sum_{i=1}^{k} (p^{(i)}-\bar{p})^2}{k-1}}} ~ , $$ where $ \bar{p} = \frac{1}{k} \sum_{i=1}^{k} p^{(i)} $. For $ k=10 $ experiments (10-fold cross-validation), the test statistic has a $t$ distribution with $ k-1=9 $ degrees of freedom. The null hypothesis can be rejected if $ |t| > t_{9,0.975} = 2.262 $ (for a two-sided test with the probability of incorrectly rejecting the null hypothesis of $0.05$).

But what if I only have the mean accuracy over all folds?

In this case, you can still perform the following test. Let $p_A$ and $p_B$ be the mean of the accuracies over $k$ folds, which are assumed to be normally distributed. If the two accuracies are independent, the difference $ p_A - p_B $ is also normally distributed. Assuming that the null hypothesis is true, we can compute the following statistic

$$ z = \frac{p_A - p_B}{\sqrt{2p(1-p)/n}} ~ , $$ where $ p=(p_A + p_B)/2 $ and number of test samples $n$. This test statistic has approximately a standard normal distribution and we can reject the null hypothesis if $ |z| > Z_{0.975} = 1.96 $.

However, this test has several problems as outlined by Dietterich (1997), e.g., $p_A$ and $p_B$ are not independent. Ideally, one should perform the proposed 5x2cv paired $t$-test, if retraining and evaluation of both models being compared is possible.

Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7), 1895-1923. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.3325&rep=rep1&type=pdf

Source Link
max0r
max0r
  • 111
  • 4

This sounds like a great application for the $k$-fold cross-validated paired $t$ test (Dietterich, 1997). However, you need the accuracies per fold of both your model and the model from the other paper. Having only the mean accuracies over all folds is not sufficient (see alternative below).

The $k$-fold cross-validated paired $t$-test

We have two classification machine learning models $A$ and $B$, that we wish to compare. Under the null hypothesis, both models should have the same test set accuracies $p_A = p_B$. Dietterich (1997) refers to error rates, which is simply $1-$accuracy. The accuracy $p$ is the probability that a randomly drawn sample from the test set will be classified correctly.

To compare both models, we are interested in the difference of the accuracies $p = p_A - p_B$. First, we perform $k$-fold cross-validation by splitting the dataset into $k$ disjoint sets of equal size. We then perform $ k $ experiments, where each set is used as test set and the remaining sets are used for training. Assuming that the $ k $ differences $ p^{(i)} = p_A^{(i)}-p_B^{(i)} $ are drawn independently from a normal distribution, we can perform Student's $ t $ test by computing the statistic

$$ t = \frac{\bar{p}\sqrt{k}}{\sqrt{\frac{\sum_{i=1}^{k} (p^{(i)}-\bar{p})^2}{k-1}}} ~ , $$ where $ \bar{p} = \frac{1}{k} \sum_{i=1}^{k} p^{(i)} $. For $ k $ experiments, the test statistic has a $t$ distribution with $ k-1 $ degrees of freedom. The null hypothesis can be rejected if $ |t| > t_{9,0.975} = 2.262 $ (for a two-sided test with the probability of incorrectly rejecting the null hypothesis of $0.05$).

But what if I only have the mean accuracy over all folds?

In this case, you can perform the following test. Let $p_A$ and $p_B$ be the mean of the accuracies over $k$ folds, which are assumed to be normally distributed. If the two accuracies are independent, the difference $ p_A - p_B $ is also normally distributed. Assuming that the null hypothesis is true, we can compute the following statistic

$$ z = \frac{p_A - p_B}{\sqrt{2p(1-p)/n}} ~ , $$ where $ p=(p_A + p_B)/2 $ and number of test samples $n$. This test statistic has approximately a standard normal distribution and we can reject the null hypothesis if $ |z| > Z_{0.975} = 1.96 $.

However, this test has several problems as outlined by Dietterich (1997), e.g., $p_A$ and $p_B$ are not independent. Ideally, one should perform the proposed 5x2cv paired $t$ test, if retraining and evaluation of both models being compared is possible.

Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7), 1895-1923. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.3325&rep=rep1&type=pdf