0
$\begingroup$

Can the Welch t-test be used with the median and MAD instead of mean and variance? I think outliers are causing problems and the median places less weight on extreme outliers.

I'm using this to test if one piece of code is faster than another. I think some of the outliers don't represent a true measurement but some kind of pausing of the process.

$\endgroup$
6
  • 1
    $\begingroup$ No. Even in principle the MAD isn't an estimator of the SD. (And on dimensional grounds alone it isn't an estimator of variance for measured variables.) And the reasons don't stop there. There'll be a better answer but it depends on your telling us more about your data. If the data are execution times then working on a logarithmic or even reciprocal scale may make sense. Can you show your data or a plot of them? $\endgroup$
    – Nick Cox
    Commented Dec 10, 2018 at 11:46
  • $\begingroup$ This is a for a general purpose algorithm to determine if one piece of code runs faster than another. There is no data available. $\endgroup$
    – user842807
    Commented Dec 10, 2018 at 13:54
  • $\begingroup$ You presumably looked at data at least loosely to know that outliers are something to worry about. Note that the point about reciprocals of times is that they are in essence speeds, e.g. twice as long means half the speed. $\endgroup$
    – Nick Cox
    Commented Dec 10, 2018 at 14:24
  • $\begingroup$ If there is no data then there is nothing to test $\endgroup$
    – Peter Flom
    Commented Dec 10, 2018 at 18:01
  • $\begingroup$ It's in principle possible to construct a test if you make some assumptions (e.g. if you assume that your data are drawn from normal distributions, you should be able to derive an approximate test in this fashion; under specific types of moderate non-normality it might continue to have reasonable properties. A similar question has been asked before, that you might find helpful: T-test using only summary data in a box plot. $\endgroup$
    – Glen_b
    Commented Dec 11, 2018 at 4:17

1 Answer 1

0
$\begingroup$

The Welch t-test is a test of the difference in means. If you wish to test the difference in medians, you can use Mood's median test or the Wilcoxon-Mann-Whitney U test.

Another possibility is to use quantile regression and treat "code piece" as a categorical independent variable. This lets you test not only the median but any quantile.

I would also try to figure out exactly what is going on with the outliers. Effort spent figuring that out will be rewarded with better data, which always helps.

$\endgroup$
3
  • $\begingroup$ Thanks. This is for a general software algorithm which runs each piece of code in turn until it can to some confidence say one is faster than the other or they are the same. Would you say Wilcoxon-Mann-Whitney U test would be a good fit for this? This iterative stopping criteria is not often seen in statistical testing. $\endgroup$
    – user842807
    Commented Dec 10, 2018 at 14:27
  • 1
    $\begingroup$ On the contrary, there is a large statistical literature on monitoring a process under headings like "quality control" or "process control". It's just that I have no detailed knowledge of what's closest to your problem, but those are search terms. $\endgroup$
    – Nick Cox
    Commented Dec 10, 2018 at 16:46
  • $\begingroup$ Many thanks for this. I've implemented the Mann-Whitney test. The last issue I have is when I was doing the Welch test repeatedly when taking measurements I needed to stop if they where still equal after some point. I did this using the difference and standard error i.e. given confidence they don't differ by 0.5%. In the case of the median I'm not sure if I should do something similar or look at soem sort of convergence test. $\endgroup$
    – user842807
    Commented Jan 16, 2019 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.