Timeline for Minimize Type II errors in black-box testing
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 26, 2018 at 7:12 | comment | added | Stephan Kolassa | The key is that you need to pre-specify your experimental setup and your analysis plan including the model. (This is often not done cleanly in clinical study sample size determination.) I am not really sure whether your question really is about how to analyse your box. If so, perhaps you could clarify. | |
| Jul 26, 2018 at 7:10 | comment | added | Stephan Kolassa | ... For instance, if you want to detect a smaller effect size ("practical bias") with given $\alpha$ and $\beta$, then $n$ will need to go up. If you want to be more stringent with your $\alpha$ level with given effect size and $\beta$, then your $n$ will need to go up. Power analysis is usually used to calculate the sample size before running an experiment, based on subjectively set $\alpha$ and $\beta$ and "relevant" effect sizes. There are power calculators for simple experiments out there, but in your case, I'd recommend writing your own in a few lines of R. | |
| Jul 26, 2018 at 7:08 | comment | added | felix91gr | There, I've now explained in more details what my problem is, since it seems I can't convey it in statistical terms. Thank you, and sorry for the inconvenience. Edit: oh, just read your comment. I think I'm getting it now, will come back after reading some more. Thanks! | |
| Jul 26, 2018 at 7:08 | comment | added | Stephan Kolassa | No problem. Wikipedia is actually pretty good at power analysis. Essentially, you have four factors playing together: (1) your sample size $n$, (2) an effect size you are interested in which I called "practical bias", (3) the probability of rejecting an unbiased box which you will accept, which is the $\alpha$ level, and (4) the probability of detecting a "practically biased" box, which is $\beta$. Power analysis analyses the interplay between these four (always in the context of a specified experimental setup and analysis). ... | |
| Jul 26, 2018 at 6:50 | comment | added | felix91gr | (I also realize that comment formatting doesn't allow for line breaks. Ouch, my last comment is kind of a pain to read) | |
| Jul 26, 2018 at 6:47 | comment | added | felix91gr | Sorry, tried to edit my comment but got cut off by the timer! Here's what I was trying to write: (Regarding the rest of your answer:) I'm sorry, I really don't understand most of that. Most importantly: - Where can I learn about power? - What tools allow me to calculate the effect size? And what is an effect in this context? Yes, I'm a noob on statistics. Sorry about that, but uni's course was pretty lame on statistical inference and I'm picking it back up after some years. | |
| Jul 26, 2018 at 6:44 | comment | added | Stephan Kolassa | Very good. Then the approach I outline should work. (And I don't understand why you write that p-value testing wouldn't work.) | |
| Jul 26, 2018 at 6:39 | comment | added | felix91gr | > You will need to accept some probability of accepting a biased box Of course! I know that I need infinite samples otherwise. I'm fine with, idk, 99% chance of rejecting a biased box. | |
| Jul 26, 2018 at 6:28 | history | answered | Stephan Kolassa | CC BY-SA 4.0 |