I'm measuring performance of 2 methods (method_A and method_B) that try to satisfy customers demands. Both methods produce results between 0 (none of the demands were satisfied) and 1 (all demands were satisfied). Also, both methods run for a certain amount of Time and, after that, they return best possible solution.

I've generated 30 datasets and solved each one with method_A and method_B with different Time limits. The results are summarized below:

enter image description here

My goal is to do some hypothesis testing with the results. I've considered to do a t-test but before that I've decided to do some normality tests. Results below:

Histogram of method_A and method_B enter image description here

Q-Q Plot and Shapiro–Wilk test for method_A enter image description here

Q-Q Plot and Shapiro–Wilk test for method_B enter image description here

For small Time values data seem to follow normal distribution so I think that t-test can be done. Am I right? However, when Time values are large the distribution seem to become non normal. This is especially visible when average results of both methods are close to 1:

  • Time equal to 80 and 90 for method_A
  • Time equal to 140 and 150 for method_B

So the question is: For these scenarios should I do Mann-Whitney U Test? For some reason, doing 2 different tests (t-test and Mann-Whitney U Test) for same data doesn't seem right...

Overall, what is the best way to do hypothesis testing with this kind of data?


I would just use the Mann-Whitney test on the raw data. From the wiki:

Unlike the t-test it does not require the assumption of normal distributions. It is nearly as efficient as the t-test on normal distributions.

In other words, even if you overcome the hurdle of determining exactly in which cases the normal assumption is valid, you don't really gain a lot of power (plus you lose the ability to compare results for different times).

  • 1
    $\begingroup$ Yeah, I'm also inclined to follow the "safety first" principle and just use Mann-Whitney test and be done with. Thanks $\endgroup$ May 17 '19 at 10:12

If you're going to treat the data for each time as a different hypothesis test, then there's probably no theoretical reason why you can't use different tests for different data sets. But practically speaking, it may be easier to use one test. Also, because the t-test and Mann-Whitney test very different hypotheses, summarizing results of these different tests can be tricky.

  • $\begingroup$ Thanks for your response. So what would you recommend? $\endgroup$ May 17 '19 at 9:56
  • $\begingroup$ Mann-Whitney would be better for what you're trying to do. $\endgroup$ May 17 '19 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.