I have one dataset which was tested under different programming languages / versions of the same algorithm. I consider them to be the treatments.

I want to test the effect of the treatments, so I collected the runtime obtained from using a treatment with the same dataset having either 50k, 100k or 1M rows, each size was processed 5 times with each treatment (same dataset same treatment 5 times), except in some cases when the bigger dataset was tested only once per treatment. This makes my data to be paired (dependent).

I selected two tests, that can be applied to non-parametric, dependent data: Wilcoxon signed rank for two samples, and Friedman's test for more than two samples.

For Wilcoxon signed rank test I plan to do it this way:

enter image description here

a) Would that be ok to use on MatLab:

[p,h] = signrank(java,python) 

b) Does it affect in anything to have 5 runtimes for the exact same dataset, then another 5 for a bigger dataset with the same information except for the IDs (e.g. it continues from ID 50 000, to 50 001, 50 0002, etc.), then 1 runtime for the 1M dataset? or is it ok to give MatLab two vectors, each one formed by the data in columns?

For Friedman's test I plan to do it this way:

enter image description here

c) Would that be ok to use on MatLab:

p = friedman(data,5)

where data is the whole matrix shown in the picture, and 5 because of the repetition of 5 runtimes per dataset size.

d) How can I address the repetition on a two-sample testing?

Final note is to say that the results of each test are going to be used separately, on an isolated way.

I struggled to reach the point in which I think I know which hypothesis testings are appropriate to the data I have, but still have some doubts on its application. Any help and comment is very much appreciated.

  • 1
    $\begingroup$ First, the differences between the runtimes are so large that you hardly need any test! Second, I don't think this is a paired design, as it is difficult to see to which commom "experimental sybjects" the treatments are applied, so you could use the independent groups test. $\endgroup$ – kjetil b halvorsen Sep 11 '16 at 10:04
  • 1
    $\begingroup$ .... if the different runs was done on different machines, with very diferent payloads, or something such, then maybe this could be a paired comparison. But nothing you told us indicates so. $\endgroup$ – kjetil b halvorsen Sep 11 '16 at 10:06

First, the runtimes are so different (about a factor ten) that there is hardly any need of a formal test! But there is still much of interest to say ...

You say you have a paired comparison, since the implementations were tested with different datasets (same datasets for each implementation, making it paired). But then, for some of the datasets, you did multiple (five) runs, making it an independent samples comparison within the paired comparison! Maybe simply those independent runs should be averaged.

Now note that formal hypothesis tests for paired comparisons typically are based on models such as $$ y_{i,1} = \mu + \epsilon_{i,1} \\ y_{i,2} = \mu + \Delta +\epsilon_{i,2} $$ where the $\epsilon$'s have some distribution we do not bother to specify, and $\Delta$ is the difference between treatments. That is, the tests depend on the asumption that the expected difference between treatments on the pairs are constant over the different experimental conditions indexed by $i$. That cannot be the case here, since runtimes would be expected to be (close to) proportional when input dataset size varies. That could be treated by a regression model, or more simply by analyzing logarithms! I leave that for you to see.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.