1
$\begingroup$

I am comparing the performance of two regression algorithms on a same dataset. I want to use t-test to evaluate whether there is a significant difference between the performance of the two algorithm.

Say there are 200 samples in total.

What I am doing right now is using a leave one out cross validation to calculate the error of each algorithm. I run the two algorithms on the 199 training samples and test the fitted regression models on the testing sample respectively for each of the 200 splits of the dataset. So I can have 200 testing squared errors for each algorithm.

At last I run t-test to compare the two 200 squared error samples.

Is there any problem with this method? Or are there any better ways to do the comparison?

I will really appreciate it if any formal reference can be provided for this specific problem.

$\endgroup$
0
$\begingroup$

As long as you are subtracting the two squared errors for each test from each other and calculating the average difference in performance you shouldn't have any issues as your sample size is big enough to assume the mean of the differences is normally distributed.

Formally you are testing whether $\bar\epsilon'^2$ is significantly different from zero. Where $\bar\epsilon'^2$ is the average difference in the squared error between your two algorithms.

Formally:

$H_{0}: \bar\epsilon'^2=0$ (null hypothesis)

$H_{A}: \bar\epsilon'^2\neq 0$ (alternative hypothesis)

Where

$\bar\epsilon'^2=\Sigma_{i}(\epsilon_{ai}^2-\epsilon_{bi}^2)/n$

and $\epsilon_{ai}^2$ is the squared error of algorithm a for on the $i_{th}$ test set and $\epsilon_{bi}^2$ is the squared error of algorithm b for that same test.

Make sure you standardise $\bar\epsilon'$ when testing.

$\endgroup$
  • $\begingroup$ Thank you! One more question: standardizing the square errors means I should divide the square errors by the deviation of the target values? $\endgroup$ – Liang Mar 23 '17 at 17:28
  • $\begingroup$ statisticshowto.com/standardized-test-statistic might help $\endgroup$ – Morgan Ball Mar 23 '17 at 17:40
  • $\begingroup$ Where $\bar\epsilon'^2$ is your sample mean and 0 is your population mean (remember your testing the hypothesis that $\bar\epsilon'^2$ is statistically different from 0 i.e. the algorithms both perform as well as each other.) $\endgroup$ – Morgan Ball Mar 23 '17 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.