Skip to main content
2 of 3
added 431 characters in body
Sextus Empiricus
  • 86.6k
  • 6
  • 115
  • 304

Make a plot, and see that these confidence intervals are very much like each other.

plot for comparison

So the two features 'indirect19' and 'indirect20' are border cases. This makes the question not only about which method is best (and all sort of details about both methods which are estimations and not exact, thus likely to slightly differ), but it also opens up the can of worms about p-values at the border of a predefined significance level (e.g. Is the exact value of a 'p-value' meaningless? or Is it wrong to refer to results as "nearly" or "somewhat" significant?).

The appropriateness of reporting only one set of p-values... In this case it doesn't really have much influence, but the behaviour is not very good. It can be considered p-hacking. People perform some experiment and, when the result is not very accurate, instead of gathering more data to be more sure, they try out many different statistical tricks to make the results look better than they really are.

It is better to just report everything that you did and represent the data for what it is, without cherry picking regarding p-values. If it is not very significant then it is just not significant. In the end, what matters is actually the effect size, and significance is just a measure of the precision in the experiment.

Also, be clear why you performed multiple tests. If you were not sure about what method is appropriate, then do not just report only a single one when the conclusions about significance suit you better. The difference in significance is not what makes the one method better than the other. If you couldn't decide which statistical method to use before doing the analysis, then you can neither after having performed the calculations. (you might figure out too later, after seeing the results that one method happened to be more powerful, but it is a slippery slope to decide afterwards based on the observed power)

Sextus Empiricus
  • 86.6k
  • 6
  • 115
  • 304