4
$\begingroup$

Having recently run an experiment, I have been left with a dataset that I don't quite know how best to handle, I think simply due to the number of independent variables to consider.

Setup

I have implemented 4 new approaches to solve a problem, which I wish to compare to an existing approach and to each other, based on their execution time, which prior to the experiment is expected to be an improvement.

To compare these approaches, they are each tested with a set of 6 case studies. Each combination of approach-casestudy is repeated 30 times, to ensure the results are representative.

All of the above is repeated for two different libraries which are used as part of the approach.

Results

In total, I am left with a total of 1800 rows (calculated as...

= (Approaches * Case Studies * Trials * Libraries)
= 5 * 6 * 30 * 2
= 1800

I believe I could validly compare the results of two approaches for a specific library and case study using the non-parametric Wilcoxon rank-sum test. However, I don't know of a testing approach I could use to determine, overall, whether one approach is better than another, or which (for the given case studies and libraries) is 'best'.

Is there some approach I could use to validly summarise the results, and therefore conclude to some level of significance which is 'best'?

Thanks, and apologies for any missing important details- I will edit whenever necessary!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
+50
$\begingroup$

Though I would have preferred to comment rather than answer, since I am no expert on non-parametric tests, I will attempt to at least help you find a suitable solution.

If I understand correctly, the 360 (6*30*2) data points per approach were generated not because of an interest in differences between case studies, trials or libraries (e.g., whether a certain approach works best for a particular case study), but simply to ensure that results are representative. It therefore appears to me that, since you seem to have used a complete block design, your main question (whether any of the approaches is better or worse) could perhaps be rephrased in terms of the Friedman test. Since post-hoc analysis for this test is available, it should then also prove possible to determine whether any approach is best (better than all the others).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the help. Further to the suggestion of the Friedman test, what means of post-hoc analysis would be suitable, which would allow the comparison of one approach against another approach, across all case studies? Or would I simply have to compare each approach-case study pair using something like a Wilcoxon? $\endgroup$
    – obfuscation
    Feb 1, 2013 at 14:44
  • 1
    $\begingroup$ @obfuscation Does this or that help any? $\endgroup$
    – Glen_b
    Feb 3, 2013 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.