Timeline for Algorithm for determining performance speedup/slowdown in a code change vs. historical data?
Current License: CC BY-SA 3.0
23 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 9, 2014 at 7:26 | comment | added | Val | You should not change your system for historic comparison. You can neither upgrade hardware nor software (OS, compilers) -- nothing for the fair comparison. Secondly, step-by-step incremental change (greedy algorithm) does not necessary leads you to the best performance :) | |
| S Jun 28, 2013 at 20:10 | history | bounty ended | CommunityBot | ||
| S Jun 28, 2013 at 20:10 | history | notice removed | CommunityBot | ||
| Jun 27, 2013 at 12:40 | answer | added | EngrStudent | timeline score: 1 | |
| Jun 27, 2013 at 12:12 | comment | added | EngrStudent | It sounds like your programs are large and so can be thought of as systems. It sounds like you are trying to improve the whole thing and not just some random part. It does you no good if "traffic" on a 1-mile stretch of "highway" goes 1000 miles an hour if the rest of the 1000 mile path has a mean speed of 40 mph. If you read books on theory of constraints, like the goal, they talk about how to find, move, and utilize "the bottleneck" for best production value. en.wikipedia.org/wiki/The_Goal_(novel) | |
| Jun 23, 2013 at 12:11 | answer | added | Jaitropmange | timeline score: 0 | |
| Jun 21, 2013 at 14:20 | comment | added | EngrStudent | I wish you could "label" the name of the nature of the change. If you had a good label, and not raw text, then you could use something like random forests to associate the label of the change to the change in the scores. For instance if you vectorize the for-loop, in MatLab (2007 era) you would get a 12x speed improvement. Instead of feeding in code with variable names, if you just gave three labels like "vectorize" "for" "mainloop" then perform the run, you would quickly get useful associations. Do you have any sample data? | |
| Jun 21, 2013 at 4:46 | answer | added | Stefan Wager | timeline score: 1 | |
| Jun 20, 2013 at 20:04 | comment | added | Michael Holman | That is correct. | |
| Jun 20, 2013 at 20:00 | history | tweeted | twitter.com/#!/StackStats/status/347806222663159809 | ||
| Jun 20, 2013 at 19:31 | comment | added | Bitwise | Just to see I understand: after you change the code, you run 400 different tests, running each of these 5 times? | |
| S Jun 20, 2013 at 19:01 | history | bounty started | Michael Holman | ||
| S Jun 20, 2013 at 19:01 | history | notice added | Michael Holman | Draw attention | |
| Jun 18, 2013 at 2:08 | history | edited | Michael Holman | CC BY-SA 3.0 |
hopefully a little more clarity
|
| Jun 17, 2013 at 21:51 | comment | added | usεr11852 | This is a really interesting question/problem. I am not fully qualified to tackle this; I think in a way your system is Markovian in the sense that all the "slow-down" effects due to a code change are compared with immediately previous condition of your system but on the other hand some changes would have a "delaying" effect regardless of the system previous state so it is not "memoryless". My first initial response would be that "naive" data aggregation is wrong. Maybe something related with Kalman filtering... I might think something and say more tomorrow. $Cool problem$! | |
| Jun 17, 2013 at 21:36 | history | edited | Michael Holman | CC BY-SA 3.0 |
hopefully clarified things a bit
|
| Jun 17, 2013 at 9:21 | comment | added | Glen_b | I suggest you use those phrases in place of the words each time, as annoying as that will probably be. At the least, define them as here. | |
| Jun 17, 2013 at 9:12 | comment | added | Michael Holman | My apologies. I'll try to clear it up in the morning. I was originally going to post this on stackoverflow when I saw a link to here and I didn't think to change my wording very much. I did not mean regression in the statistical sense but in the software definition. To use more general terms, regression=test ran slower, improvement =test ran faster. A checkin is a code change that a developer submits. I run tests using versions of our program that are built after every new piece of code is submitted, and I want to find when a code change has actually caused a test to actually run slower. | |
| Jun 17, 2013 at 5:26 | review | Close votes | |||
| Jun 18, 2013 at 20:05 | |||||
| Jun 17, 2013 at 5:04 | comment | added | Glen_b | Lots of unexplained jargon/abbreviations (I fixed some but stuff like 'dev' - is that 'developer', or some other thing?), the use of statistical jargon like 'regression' to mean something else without clearly flagging it as such; and lack of a clearly defined question all combine to make this a very hard question to answer. We're not software developers. | |
| Jun 17, 2013 at 5:03 | history | edited | Glen_b | CC BY-SA 3.0 |
expand abbreviations, other clarifications
|
| Jun 17, 2013 at 2:51 | review | First posts | |||
| Jun 17, 2013 at 5:32 | |||||
| Jun 17, 2013 at 2:31 | history | asked | Michael Holman | CC BY-SA 3.0 |