Mean classification accuracy and the standard deviation on 44 cases of Method 1 are 0.7857 and 0.1076, respectively.

Mean classification accuracy and the standard deviation on 44 cases of my method are 0.8081 and 0.0711, respectively.

So the mean classification performance of my method is better than that of Method 1. But the p-value between Method 1 and my method is 0.073, which is a little larger than the 0.05.

How to improve my results? Thanks!


You don't need to "improve your results." That sounds like p-hacking, i.e. nudging the p-value below 0.05. Don't focus on the p-value. There's nothing magical about p = 0.05. What you should focus on is communicating your data. In this case the data says that your method is slightly better than Method 1. However there's probably more here, and some of it might be interesting.

One thing to look at is whether classification accuracy is a valid measure. Are the classes imbalanced? If so, accuracy might not be useful, since a dumb classifier could simply guess the more abundant class for >50% accuracy.

What about the data points that Method 1 fails on? Does your classifier do better than average on these? If so, that could be interesting. Or does your classifier tend to fail in the same way that Method 1 fails?

What data set are you using to compare the classifiers? Can you collect or simulate more data? How do the classifiers compare for those data sets?

If nothing interesting comes up, then you might just want to conclude with the information you gave in your answer: your classifier does slightly better than Method 1.

  • $\begingroup$ Hi, thank you very much for so detail answer! In fact, in 17 cases of the 44 cases, my method is worse than the Method 1, and in the other 27 cases my method can achieve better results. In addition, in the 17 cases, some cases are a little inferior on my method than Method 1, and some other cases are a relatively big inferior than Method 1, so I think those bad cases made the p-value large. $\endgroup$ – mining May 19 '17 at 23:29

The quality of your study has nothing to do with the p-value; it cannot tell you that you need to "improve" your results.

You may be underpowered and need more participants to get the effect that you are looking for. You can try replicating the study again, with a larger sample size, and then doing a small-scale meta-analysis between the two studies. Your effect size, in Cohen's d terms, is 0.25. If the actual effect size is 0.25, you would need 253 people per group to get a significant result, p < .05, 80% of the time.

As I'm typing this out, @Qroid has already pointed out the p-hacking nature of this question. So in addition to what I'm saying, I also agree that it sounds like p-hacking. If you are unfamiliar with the term, here is a first paper and a second paper that I enjoy on the topic. And a third on small-scale meta-analyses and underpowered studies.

Do not decide something needs to be done based on a p-value.

EDIT: I didn't fully understand what you meant by mean classification, (I'm a psychologist by training, so I assumed you were talking about results to some type of cognitive task), so the Cohen's d bit above might be a little off.

  • $\begingroup$ Hi, sir, thank you very much for your kind answer and valuable suggestions on meta-analysis and the papers. In fact, I'm handling the medical image segmentation problem. I need to segment the tumors using computer algorithms. For each case in the whole 44 cases, there is a ground truth labeling generated by doctors. And my method or Method 1 can give a prediction for each case, and compared to the ground truth, we get a classification accuracy/metric to measure how well the method is. Then we can get the mean classification value for each method on these 44 cases. $\endgroup$ – mining May 19 '17 at 23:42
  • $\begingroup$ Now my method seems a little better than Method 1 when considering the mean accuracy. But the p-value is larger than 0.05, so from the viewpoint of statistical aspects, the results is not significant, which means my method is not better than the Method 1. So this is sad for me... $\endgroup$ – mining May 19 '17 at 23:43

You don't improve your results. You are done. That is the whole point of a p-value. Through random effects alone you could drive the value below .05 by carefully selecting data, but that isn't what you are trying to do.

If I am interpreting your statements correctly, you are comparing your method to an existing method, and the difference does not appear to be different from zero to a 5% confidence level. Fundamentally, this is saying that your method is the other method in disguise.

This is valuable information. What if the other method happens to match nature perfectly? Then it is intrinsically the best method, and all others are inferior. You wouldn't want a false positive regarding your method.

If I needed a "significant" result, I would go back into the literature to see what others had done and why they did it. You may find a better method in the literature, or if you took multiple items from the literature, you might find a better combination of variables.

It is important to remember that a finding of no significance is a finding that is important.

  • $\begingroup$ Hi, thank you very much for your detail answer and kind suggestions! Indeed, I'm struggling to improve my method over an existing method. I eager to beat that method because without that the paper cannot be published. $\endgroup$ – mining May 19 '17 at 23:36
  • 1
    $\begingroup$ That reads as if you're saying "without p-hacking the paper cannot be published" (with the converse implication that whatever journal you're trying to get into will happily publish a paper if it has a small-but-meaningless p-value in it). Is that really the state of things? $\endgroup$ – Glen_b -Reinstate Monica May 20 '17 at 2:11
  • $\begingroup$ @Glen_b my training is in social psychology. My answer to that question is: yes. One of the many reasons I'm leaving academia after PhD $\endgroup$ – Mark White May 20 '17 at 4:31
  • 1
    $\begingroup$ I'm curious to see which journals you publish in. Every Psych association with published editorial policy and each journal I can locate right now says almost the exact opposite in its policies on reporting statistical analysis. In social psychology one journal went as far as to ban p-values, to my recollection. The American Psychological Association has been railing against such practices for decades now. $\endgroup$ – Glen_b -Reinstate Monica May 20 '17 at 5:35

There is a move in some quarters not even to use p values. First, there is nothing "scientific" about .05 rather than .049 or .1 rather than .099 or whatever. The levels used are rules of thumb created in the (I think) early 20th century because researchers saw those as reasonable numbers to test null hypothesis with.

Second, people confuse these with having strong effect sizes, when they don't mean that at all. Just because your p score is .0001 does not mean you have a "substantively" important effect size. The strength of the effect size is a judgement one has to make based on your understanding of the phenomenon.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.