0
$\begingroup$

I'm working on comparing 2 algorithms with an experimental protocol that produce 100 folds for each one.

As a result, I found that my algorithm got (49.29 $\pm$ 1.69) and the baseline got (50.40 $\pm$ 2.16). I applied ANOVA and other tests and I always got a p-value of 0.60.

Method: Deep learning.

Goal: comparing 2 algorithms (mine and another)

Field: computer-vision

Hypothesis ($\alpha=0.05$):

  • $H_0$: the mean of the results are equal.
  • $H_a$: the mean of the results are unequal. (advantage go to the adversary)

Results:

  • $mean_{proposed}$ < $mean_{baseline}$

  • Population: 2 ( proposed and baseline)

  • sample size = 100

  • $P=0.6$ and $\alpha = 0.05$

  • $P > 0.05 $ $=>$ no significant difference

My conclusion : the 2 algorithms are equal.

accuracy distribution: bleu: my model, Green: baseline

Can a reviewer reject my conclusion (my fail to reject the H0)?

How can I defend my point of view?

If you need more information, please ask them in a comment.

$\endgroup$
  • $\begingroup$ What are you trying to show by this test? That your algorithm does not produce results significantly better than the other algorithm? $\endgroup$ – Jan Kukacka Apr 9 at 13:07
  • $\begingroup$ I wanna prove that the 2 algorithms are similar. $\endgroup$ – zeronoid Apr 9 at 13:36
0
$\begingroup$

If you are comparing the two algorithms on the same data folds, than you can use paired t-test to evaluate if they produce similar results. Also, if you want to show that they are similar (opposed to "one is better than the other"), two-sided test would make more sense.

Finally, if you get large p-values in this test, it does not mean you accept the null hypothesis. It means that you don't have sufficient evidence to reject it. Make sure you interpret the results of your test correctly and dont't infer something that is actually not true. See e.g. Why do statisticians say a non-significant result means "you can't reject the null" as opposed to accepting the null hypothesis?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.