Statistical test for cross validation with a low mean

Question

I'm working on comparing 2 algorithms with an experimental protocol that produce 100 folds for each one.

As a result, I found that my algorithm got (49.29 $\pm$ 1.69) and the baseline got (50.40 $\pm$ 2.16). I applied ANOVA and other tests and I always got a p-value of 0.60.

Method: Deep learning.

Goal: comparing 2 algorithms (mine and another)

Field: computer-vision

Hypothesis ($\alpha=0.05$):

• $H_0$: the mean of the results are equal.
• $H_a$: the mean of the results are unequal. (advantage go to the adversary)

Results:

• $mean_{proposed}$ < $mean_{baseline}$

• Population: 2 ( proposed and baseline)

• sample size = 100

• $P=0.6$ and $\alpha = 0.05$

• $P > 0.05 $ $=>$ no significant difference

My conclusion : the 2 algorithms are equal.

Can a reviewer reject my conclusion (my fail to reject the H0)?

How can I defend my point of view?

If you need more information, please ask them in a comment.

Answer 1

If you are comparing the two algorithms on the same data folds, than you can use paired t-test to evaluate if they produce similar results. Also, if you want to show that they are similar (opposed to "one is better than the other"), two-sided test would make more sense.

Finally, if you get large p-values in this test, it does not mean you accept the null hypothesis. It means that you don't have sufficient evidence to reject it. Make sure you interpret the results of your test correctly and dont't infer something that is actually not true. See e.g. Why do statisticians say a non-significant result means "you can't reject the null" as opposed to accepting the null hypothesis?