I have one dataset which was tested under different programming languages / versions of the same algorithm. I consider them to be the treatments.
I want to test the effect of the treatments, so I collected the runtime obtained from using a treatment with the same dataset having either 50k, 100k or 1M rows, each size was processed 5 times with each treatment (same dataset same treatment 5 times), except in some cases when the bigger dataset was tested only once per treatment. This makes my data to be paired (dependent).
I selected two tests, that can be applied to non-parametric, dependent data: Wilcoxon signed rank for two samples, and Friedman's test for more than two samples.
For Wilcoxon signed rank test I plan to do it this way:
a) Would that be ok to use on MatLab:
[p,h] = signrank(java,python)
b) Does it affect in anything to have 5 runtimes for the exact same dataset, then another 5 for a bigger dataset with the same information except for the IDs (e.g. it continues from ID 50 000, to 50 001, 50 0002, etc.), then 1 runtime for the 1M dataset? or is it ok to give MatLab two vectors, each one formed by the data in columns?
For Friedman's test I plan to do it this way:
c) Would that be ok to use on MatLab:
p = friedman(data,5)
where data is the whole matrix shown in the picture, and 5 because of the repetition of 5 runtimes per dataset size.
d) How can I address the repetition on a two-sample testing?
Final note is to say that the results of each test are going to be used separately, on an isolated way.
I struggled to reach the point in which I think I know which hypothesis testings are appropriate to the data I have, but still have some doubts on its application. Any help and comment is very much appreciated.