Timeline for Determine the sample size to validate a bug fix
Current License: CC BY-SA 4.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 16, 2023 at 19:23 | comment | added | dipetkov | Is there a bug that has been detected and fixed (in which case the software engineering approach to testing is to write appropriate unit tests to verify this; if you've fixed a bit then you know what was causing it)? Or the functionality that is being tested is non-deterministic/stochastic? There are approaches for this kind of unit testing as well; see for example this blog post. I don't see how Null Hypothesis Significance Testing offers any advantages over proper unit testing. | |
| Apr 16, 2023 at 16:59 | comment | added | buckley | Are you asking about a situation where you do exactly 1000 trials in each test run and do multiple test runs of 1000 trials? > That is what I conceptually want it seems. But I am here to find shortcuts right? :) That is why I thought of a power analysis where you find the minimum sample size. Executing 1000*1000 = 1 million tests is something that is to be avoided (time/energy/cost) and since I ask for x% certainty/confidence it seems that 1M executions can be made lower in function of that confidence. | |
| Apr 16, 2023 at 15:42 | comment | added | EdM | When you say things like "what is the chance it fails 99% of the time 0 out of 1000?" are you asking about a situation where you do exactly 1000 trials in each test run and do multiple test runs of 1000 trials? | |
| Apr 16, 2023 at 8:40 | comment | added | buckley | @GreggH I updated my question as I am as well interested in showing that the previous failure rate is gone with a certain confidence. | |
| Apr 16, 2023 at 8:34 | history | edited | buckley | CC BY-SA 4.0 |
added 400 characters in body
|
| Apr 15, 2023 at 17:57 | history | became hot network question | |||
| Apr 15, 2023 at 16:35 | answer | added | EdM | timeline score: 3 | |
| Apr 15, 2023 at 16:24 | comment | added | Gregg H | Have you considered the possibility that when the bug is indeed fixed, ¿there might be zero % chance of an error occurring? I ask this, because in this case, a distributional analysis could be tricky. | |
| Apr 15, 2023 at 16:09 | comment | added | buckley | Does that mean that you would like to estimate an upper limit to the frequency of bug occurrence after the fix? > I would like to determine a sample size so I could say that I am 99% sure (or any other number) the bug is fixed. Is that a reasonable question or is it faulty? The failure rate is 3 out of 1000, so 0.003, and made the correction. Thank you | |
| Apr 15, 2023 at 16:08 | history | edited | buckley | CC BY-SA 4.0 |
added 1 character in body
|
| Apr 15, 2023 at 15:04 | comment | added | EdM | Please edit the question to provide more details about what you mean by "I would like to validate a bug fix so that it doesn't appear within a degree of confidence." Does that mean that you would like to estimate an upper limit to the frequency of bug occurrence after the fix, or something else? Also, do you mean 3 out of 100 test runs or 3 out of 1000 (which would be consistent with your use of 0.003). Note that software-specific questions are off-topic on this site, but there does seem to be some on-topic statistical content here. | |
| S Apr 15, 2023 at 9:47 | review | First questions | |||
| Apr 15, 2023 at 10:01 | |||||
| S Apr 15, 2023 at 9:47 | history | asked | buckley | CC BY-SA 4.0 |