# Is my better score with a different machine learning model statistically significant?

I have a question concerning the p-value and how you can say that this selection of features is better than that other one.

To make things simple let's assume that I'm working on the spam classification problem.

I've tested two different ways to do my feature engineering :

OPTION 1 = bag of words
Accuracy : 90%

OPTION 2 = tf-idf
Accuracy : 91%

Those results seem pretty close with an edge for tf-idf.
However this could be caused by the random seed, the dataset I trained my algorithm on, the dataset I tested my data on etc.

My question is : can I say that OPTION2 is better than OPTION1 ?

I looked on Wikipedia for information on statistical significance and null hypothesis and from what I've understood I need to compute the probabilty of my test happening if OPTION2 is better than OPTION1 (H0)

How can I do this ?

And there seems to be a lot of controversy around the p-value (how do you set the threshold ? the fact that it is mistaken with the probability of the null hypothesis being true etc) so in the end am I not better off running my algorithm different times with different random seeds and on different parts of the dataset and do some sort of average on those values ?

I would not look for a p-value to start with. First, how did you assess the accuracy of your models ?

I will assume you obtained the performance using cross validation. Then, you also have access to the standard deviation of the accuracy (that can be seen as a random value as well). Per example, these two situations are completely different :

Situation 1 :

• tf-idf $(90\%, 90\%,91\%,90\%,89\%)$
• Bag of words $(90\%, 92\%,91\%,91\%,91\%)$

Situation 2 :

• tf-idf $(80\%, 100\%,95\%,90\%,85\%)$
• Bag of words $(90\%, 92\%,94\%,88\%,91\%)$

I would be more confident that features generated using a Bag of words approach produce a better model than these generated using tf-idf in situation 1 than in situation 2.

Now, if you want to put this in a statistical significance test, you could do the following :

Calling $S_{tf-idf}$ the random value representing the accuracy of the model obtained using a tf-idf preprocessing and $S_{BoW}$ the random value representing the accuracy of the model obtained using a bag of words preprocessing, the hypothesis would be :

$$\mu_{BoW}=\mathbb{E}[S_{BoW}]>\mu_{tf-idf}=\mathbb{E}[S_{tf-idf}]$$

Which turns out to be sample means comparison.

• I really like your answer, this was what I was leaning towards. Didn't know about sample means comparison but its exactly what I'm looking for, thanks a lot ! Commented Jun 21, 2017 at 15:03