2
$\begingroup$

I perform different rank-based non-parametric tests (Friedman and Nemenyi) post-hoc and noticed that the decimal places of the samples have a strong effect on their power. Please consider the following toy example:

Samples 1: 0.12,0.23,0.22,0.17
Samples 2: 0.15,0.17,0.18,0.1
Samples 3: 0.19,0.29,0.27,0.19

Using all decimal places might show that samples 1&2 are not significantly different but samples 1&3 and 2&3 are. Nevertheless, omitting the last decimal place will make samples 1 equal to samples 3.

Thus, omitting too many decimal places might make the samples too similar (they will share a rank). On the contrary, not omitting them makes them prone to small irregularities in sample computation or handling of floats (every sample gets a different rank, although being very close to each other).

Questions:

  1. What is the best way to deal with this, especially in a publication?
  2. Is the difference between per samples described as "effect size"? In this case, rounding to a specific amount of decimal points would serve as a constraint to the minimum effect size to be considered.

Thanks!

Improve this question
$\endgroup$
2
  • $\begingroup$ What do you mean by a "strong effect on ... power"? What were the p values when you conducted the statistical tests both ways? $\endgroup$
    – Joel W.
    Commented Aug 19, 2014 at 17:56
  • $\begingroup$ As a wrote, this was a "toy example". Thus, there are no meaningful p-values to report, I'm afraid. The point is that the test may stick with the null hypothesis although it is false (type 2 error) for two decimal places when omitting the last one. Thus, omitting decimal places may give the test a lower power. $\endgroup$
    – Chris
    Commented Aug 20, 2014 at 7:08

1 Answer 1

Reset to default
1
$\begingroup$

The "best way to deal with this" depends on whether your measurements have meaning to 2 decimal points. For example, do you think a score of 0.10 is meaningfully different from a score of 0.12? As an analogy, if I measured the weight of adult males in pounds to two decimal places in a study of weight loss due to exercise, I would suggest that there is no meaningful difference between a weight loss of 0.10 and 0.12 pounds - since most scales are not sensitive to that level and most people are interested in weight loss of 10+ pounds. If statistical analyses I did with 1 and 2 decimal places produced a change in statistical significance, I would not trust the finding.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.