# Is my hypothesis testing approach correct?

I have one dataset which was tested under different programming languages / versions of the same algorithm. I consider them to be the treatments.

I want to test the effect of the treatments, so I collected the runtime obtained from using a treatment with the same dataset having either 50k, 100k or 1M rows, each size was processed 5 times with each treatment (same dataset same treatment 5 times), except in some cases when the bigger dataset was tested only once per treatment. This makes my data to be paired (dependent).

I selected two tests, that can be applied to non-parametric, dependent data: Wilcoxon signed rank for two samples, and Friedman's test for more than two samples.

For Wilcoxon signed rank test I plan to do it this way:

a) Would that be ok to use on MatLab:

[p,h] = signrank(java,python)


b) Does it affect in anything to have 5 runtimes for the exact same dataset, then another 5 for a bigger dataset with the same information except for the IDs (e.g. it continues from ID 50 000, to 50 001, 50 0002, etc.), then 1 runtime for the 1M dataset? or is it ok to give MatLab two vectors, each one formed by the data in columns?

For Friedman's test I plan to do it this way:

c) Would that be ok to use on MatLab:

p = friedman(data,5)


where data is the whole matrix shown in the picture, and 5 because of the repetition of 5 runtimes per dataset size.

d) How can I address the repetition on a two-sample testing?

Final note is to say that the results of each test are going to be used separately, on an isolated way.

I struggled to reach the point in which I think I know which hypothesis testings are appropriate to the data I have, but still have some doubts on its application. Any help and comment is very much appreciated.

• First, the differences between the runtimes are so large that you hardly need any test! Second, I don't think this is a paired design, as it is difficult to see to which commom "experimental sybjects" the treatments are applied, so you could use the independent groups test. – kjetil b halvorsen Sep 11 '16 at 10:04
• .... if the different runs was done on different machines, with very diferent payloads, or something such, then maybe this could be a paired comparison. But nothing you told us indicates so. – kjetil b halvorsen Sep 11 '16 at 10:06

Now note that formal hypothesis tests for paired comparisons typically are based on models such as $$y_{i,1} = \mu + \epsilon_{i,1} \\ y_{i,2} = \mu + \Delta +\epsilon_{i,2}$$ where the $\epsilon$'s have some distribution we do not bother to specify, and $\Delta$ is the difference between treatments. That is, the tests depend on the asumption that the expected difference between treatments on the pairs are constant over the different experimental conditions indexed by $i$. That cannot be the case here, since runtimes would be expected to be (close to) proportional when input dataset size varies. That could be treated by a regression model, or more simply by analyzing logarithms! I leave that for you to see.