Significant Statistical difference One-sample t-test I am trying to test if one algorithm is statistically significantly faster than another using the One-sample t-test
These are the results I have and I am trying to prove Algo2 is significantly faster than Algo1
        Mean (ms)  Variance
Algo1     5171       782 
Algo2     3753       1920

with a sample size of 78,869
I've tried using the equation:

but I don't seem to be getting a value that makes sense.
How do I work this out?
 A: If you want to calculate a statistic then a two sample t-test will work here.  It's the difference between your means over the standard error of the difference.  The standard error of the difference is $\frac{\sqrt{var1 + var2}}{n}$. n = sample size.
If that t value you calculate is over 2 and your n is over 60 it's statistically significant.
OK, so that's if you want to actually calculate the stat.  I hope you can see though, as n gets large you can make the standard error of the difference arbitrarily small.  So, it's perfectly reasonable to just say they are different.  The test is kind of pointless in this case.  When you can run 10s of thousands of iterations of the algorithm then there's no point in calculating a t-test.  You can make assertions like you know the population (within your machine or set of machines you tested).  Statistical tests like that are for small samples.
In your case a compelling demonstration would be with two histograms of the samples on a single plot.  They likely won't even overlap at all.  That's much more meaningful because with your very large sample size the two distributions could overlap a lot and the t-test still find a 'significant' difference.  Don't substitute statistically significant for meaningful.
The more interesting thing to be discussing is not whether algo 2 is really faster; it obviously is.  It's whether the difference is worth whatever the other differences between algo 1 and algo 2 are.
A: I don't see how you can prove Algo2 is significantly faster than Algo 1 using one-sample t-test. If you know or can assume that Algo1 and Algo2 are independent of each other and have equal variance, you can definitely prove the difference between them using two-sample t-test.
Two sample t-test:


*

*Independent two-sample t-test (equal sample sizes and equal variance) is only used when both (1) two sample sizes (the number of Algo1 and Algo2) are equal AND (2) can be assumed that the two distributions have the same variance.

*Independent two-sample t-test (unequal sample sizes and equal variance) is only used when it can be assumed that the two distributions have the same variance.

*Independent two-sample t-test (unequal sample sizes and unequal variance)

*Dependent t-test is used when the samples are dependent (e.g. repeated measure).


Wikipedia has a nice explanation of them.
