Timeline for Algorithm for determining performance speedup/slowdown in a code change vs. historical data?
Current License: CC BY-SA 3.0
4 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 27, 2013 at 12:03 | comment | added | EngrStudent | Good statistical process control looks at both the mean and the standard deviation. If you have a code change that substantially changes the variability of the run-time, that can be a significant change. itl.nist.gov/div898/handbook/pmc/pmc.htm | |
| Jun 25, 2013 at 22:19 | comment | added | Jaitropmange | I was thinking you cared most about total time but re-reading your question I see your focus is the individual test times. But that is fine, the code I posted still works for that. Just ignore the total time t-test and look at the t-stats for the individual tests. By using the individual test historical variation (the d_stdev variables in the R code) it accounts for differences in order of magnitude. | |
| Jun 25, 2013 at 19:03 | comment | added | Michael Holman | This is interesting, but I don't think it is workable for us. We can't just add up total time. Since some tests might speed up and others might slow down and they could cancel each other out. Also, not all are on the same order of magnitude. Some tests run in 1ms, and others take 1000ms. So a change in the 1ms test case would almost certainly be overshadowed by variance from the 1000ms testcase. | |
| Jun 23, 2013 at 12:11 | history | answered | Jaitropmange | CC BY-SA 3.0 |