0
$\begingroup$

I am trying to test if one algorithm is statistically significantly faster than another using the One-sample t-test

These are the results I have and I am trying to prove Algo2 is significantly faster than Algo1

        Mean (ms)  Variance
Algo1     5171       782 
Algo2     3753       1920

with a sample size of 78,869

I've tried using the equation:

one-sample t-test

but I don't seem to be getting a value that makes sense.

How do I work this out?

$\endgroup$
  • $\begingroup$ @Procrastinator I do not have R $\endgroup$ – Chris Apr 18 '12 at 19:02
  • 1
    $\begingroup$ OK. In addition that test statistic you are using is not the one you should use to compare two samples. You are using the one sample t-test when you actually need the two sample t test. $\endgroup$ – user10525 Apr 18 '12 at 19:04
  • $\begingroup$ Is the sample size you're reporting for both tests combined or an amount that you collected for each test. $\endgroup$ – John Apr 18 '12 at 19:09
  • 2
    $\begingroup$ Your data show that timings for algo1 tend to be around $5171$ give or take $\sqrt{782}=28$ while timings for algo2 tend to be around $3753$ give or take $\sqrt{1920}=44$. The mean timings are over $1400$ ms apart, which is more than an order of magnitude greater than the typical variation. Such a difference is so blatantly obvious that no further testing is needed. $\endgroup$ – whuber Apr 18 '12 at 19:22
4
$\begingroup$

If you want to calculate a statistic then a two sample t-test will work here. It's the difference between your means over the standard error of the difference. The standard error of the difference is $\frac{\sqrt{var1 + var2}}{n}$. n = sample size.

If that t value you calculate is over 2 and your n is over 60 it's statistically significant.

OK, so that's if you want to actually calculate the stat. I hope you can see though, as n gets large you can make the standard error of the difference arbitrarily small. So, it's perfectly reasonable to just say they are different. The test is kind of pointless in this case. When you can run 10s of thousands of iterations of the algorithm then there's no point in calculating a t-test. You can make assertions like you know the population (within your machine or set of machines you tested). Statistical tests like that are for small samples.

In your case a compelling demonstration would be with two histograms of the samples on a single plot. They likely won't even overlap at all. That's much more meaningful because with your very large sample size the two distributions could overlap a lot and the t-test still find a 'significant' difference. Don't substitute statistically significant for meaningful.

The more interesting thing to be discussing is not whether algo 2 is really faster; it obviously is. It's whether the difference is worth whatever the other differences between algo 1 and algo 2 are.

$\endgroup$
  • $\begingroup$ So I would have (5171-3753)/SQRT((782+1920)/78869) = 7661.022? So I can say that this is definitely significant? $\endgroup$ – Chris Apr 18 '12 at 19:13
  • 2
    $\begingroup$ You can probably take another tack and say it's definitely pointless to calculate significance.... but yeah, that's an unlikely outcome to have occurred if there really were no difference in speed... which is what significance means in this case. It's an unwieldy way to be discussing things that you can avoid if you just avoid the statistic. Think of things like a t-test as a crutch for when you don't have better options. You have lots of better options. $\endgroup$ – John Apr 18 '12 at 19:19
  • $\begingroup$ thanks :) I thought I could just say that it is significant because of the sample size but I think its more showing the fact I know how to calculate significance if I need to $\endgroup$ – Chris Apr 18 '12 at 19:23
  • $\begingroup$ ahh sorry didn't realise, its not technically the homework, just helps to do it $\endgroup$ – Chris Apr 18 '12 at 19:38
1
$\begingroup$

I don't see how you can prove Algo2 is significantly faster than Algo 1 using one-sample t-test. If you know or can assume that Algo1 and Algo2 are independent of each other and have equal variance, you can definitely prove the difference between them using two-sample t-test.

Two sample t-test:

  • Independent two-sample t-test (equal sample sizes and equal variance) is only used when both (1) two sample sizes (the number of Algo1 and Algo2) are equal AND (2) can be assumed that the two distributions have the same variance.
  • Independent two-sample t-test (unequal sample sizes and equal variance) is only used when it can be assumed that the two distributions have the same variance.
  • Independent two-sample t-test (unequal sample sizes and unequal variance)
  • Dependent t-test is used when the samples are dependent (e.g. repeated measure).

Wikipedia has a nice explanation of them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.