0
$\begingroup$

I am testing a model that we developed against another model for error generation. We hypothesized that our model generates less error.

To statistically test our hypothesis we applied student's t-test. We took the magnitude of error generated by our model and the other model. Our null hypothesis was that there was no difference in the performance of our model and the other model. We rejected the null hypothesis by getting a small p-value (<0.05). Did I do it right?

$\endgroup$
1
  • 2
    $\begingroup$ To get useful answers, you will need to provide much more details. For instance: what is a model for error generation? Did you estimate the models on some data? How do you define "magnitude of error"? Just to name a few. $\endgroup$ – Michael M Mar 24 '16 at 7:48
1
$\begingroup$

A priori I would say that the method is valid. Only a few points:

The t-test requires, to be justified by the theory, that your error measurements follow a normal distribution. There exist tests to verify that, but a plot is usually sufficient to convince oneself. Otherwise, look at non-parametric tests (e.g. Mann-Withney).

Then the test measures if the respective averages are different between the two methods, taking their variance into account.

At 0.05 p-value threshold, it means that you still have a 5% chance that the two methods generate errors according to the same distribution, i.e. the new model is no improvement. Why use a threshold anyway ? Read the p-value as it gets out of the test. It gives you the probability that you a are wrong, which you can base your decision on. Always report it, anyway.

$\endgroup$
1
  • $\begingroup$ I don't see any reason to suppose absolute errors will be approximately normal. I'd tend to expect them to be right skew. $\endgroup$ – Glen_b Mar 26 '16 at 6:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.