I have the following accuracy results of 10 fold cross validation for different machine learning algorithms, My question is how can I identify whether the algorithm results are statistically significant? And how can I measure their p-value.

I have asked a similar question here I realise that I literally don't know anything about the statistical terms, p-values, ttest etc...

So I am kindly asking you, either with python or matlab, how I can prove their are statistically significant.

Thank you soooo much.

SVM=[0.88571429, 0.85714286, 0.92753623, 0.92647059, 0.94029851, 0.94029851, 0.94029851, 0.94029851, 0.92307692, 0.95384615]
ExtraTrees=[0.91428571, 0.92857143, 0.86956522, 0.92647059, 0.89552239, 0.91044776, 0.91044776, 0.95522388, 0.95384615, 0.95384615]
LogisticRegression=[0.92857143, 0.84285714, 0.92753623, 0.94117647, 0.94029851, 0.92537313, 0.92537313, 0.91044776, 0.90769231, 0.95384615]
RandomForest=[0.88571429, 0.88571429, 0.91304348, 0.89705882, 0.92537313, 0.95522388, 0.91044776, 0.91044776, 0.92307692, 0.95384615]
KNN=[0.87142857, 0.82857143, 0.89855072, 0.92647059, 0.91044776, 0.89552239, 0.88059701, 0.89552239, 0.84615385, 0.92307692] 
  • $\begingroup$ Statistical significance relates to a specific hypothesis test. What's the hypothesis and test statistic? $\endgroup$ – Glen_b -Reinstate Monica Oct 17 '16 at 1:21
  • $\begingroup$ We want to show the accuracy result of these five different classifiers are significantly different. So, we cannot just pick one of these classifier and use it, since their result accuracy are not close (insignificant). This is what we aim to show. $\endgroup$ – Memin Oct 17 '16 at 2:24

It appears that you are not aware of what cross validation is and how it differs from model selection.

When one builds or selects a model, that is when you check if the model coefficients are significant. So in your case, you would have to go back to step one where you built each model and look at the model summary to find your p-value.

When you do cross validation your are verifying if the model you have selected performs fairly similarly irrespective of which training data subset it was built on.

However, given how your awareness of these concepts are limited, I'd suggest first learning the statistical model building process and then try out the more advanced techniques like k fold cross validation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.