Timeline for How to supplement comparing benchmark results (means) with a two-sided t-test?
Current License: CC BY-SA 3.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 18, 2013 at 18:37 | comment | added | russellpierce | For your purposes 7 decimal places picks up variance that is so small you don't care about it. You say that when you round to 5 decimal places all of the values come up the same. Do you care about performance differences in the 6th decimal place? If not, round to 5. Do you care about performance differences in the 5th decimal place, if not, round to 4 ... and so on. I don't quite understand what measurement unit your mean is in... but it doesn't make much sense to record a number of decimal places for your sample that exceeds twice the resolution of your timer. | |
| Feb 18, 2013 at 18:34 | comment | added | russellpierce | @John-DavidDalton: Statistical tests are insensitive to whether something is practically significant. They only are sensitive to what is statistically significant. For example, tests of normality frequently report non-normality with large sample sizes even when the underlying distribution is practically normal. You have two possible solutions to adjusting how sensitive the t-test is. One is to adjust the $\alpha$ value. E.g., you could arbitrarily decide that only p < 9.865876e-10 is statistically significant. Two is to round to a lower number of decimal places. | |
| Jan 23, 2012 at 21:45 | comment | added | Michael Lew | I'm not sure that you are really considering what statistical tests really do, but if you want a test that will deal with a median rather than the mean (medians are less affected by the magnitude of extreme values within the sample), then you could consider a non-parametric test in place of Student's t-test. Try a permutations test on the medians, or a Wilcoxon (Mann-Whitney) test. | |
| Jan 23, 2012 at 3:29 | comment | added | John-David Dalton |
I would like to assume that the tests are reasonably close, in this case identical tests seem to fall within ~5% of each others means. But the t-test is too picky with what it considers statistically significant when comparing benchmark results. Is there another test I can use that is less sensitive to the differences in mean value? Is the fact that the mean values are so small, something like 0.0001135, a problem?
|
|
| Jan 22, 2012 at 23:00 | history | answered | Michael Lew | CC BY-SA 3.0 |