Timeline for Increasing sample size to obtain width of CI / SE: p-hacking?
Current License: CC BY-SA 4.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 21 at 23:43 | vote | accept | Jeremy Miles | ||
| Aug 24, 2021 at 16:53 | comment | added | Jeremy Miles | That's a good idea. Thanks. | |
| Aug 24, 2021 at 6:41 | comment | added | Glen_b | Why not take account of the error (i.e. the fact that its a random variable not a known population quantity) in the pilot estimate of variance when forming the coverage calculation? If you have a model, forming a pivot and calculating coverage is at least amenable to simulation at the pilot stage, or you could take a bootstrapping approach in large samples. | |
| Aug 24, 2021 at 2:16 | history | became hot network question | |||
| Aug 23, 2021 at 22:15 | comment | added | Jeremy Miles | Indeed - some might say that about p-values themselves. :) I don't think I've lost objectivity - I want to see if there's a problem with this approach, 'cos if there is I wont' do it (and say sorry to customer). | |
| Aug 23, 2021 at 21:26 | comment | added | BruceET | Sometimes a bad idea is so tempting that excuses abound and objectivity is lost. | |
| Aug 23, 2021 at 21:22 | comment | added | Jeremy Miles | @BruceET - that discussion seems to be mixing up discussion of p-values, effect sizes and CIs/SEs. All I'm looking at is SE, so I'm not sure I need a sequential approach. (Which I would if looking at p). | |
| Aug 23, 2021 at 21:11 | comment | added | BruceET | This recent Q & A discusses why a sequential approach such as as yours is inappropriate if you use standard tests (or formulas for CIs). [Strictly speaking I guess it can't be "P-hacking" unless you're using P-values.] // In your work a sequential approach might be useful. However, if you do use a sequential approach, then you need to use methods that take your particular sequential scheme into account. (Hence @Lewian's suggested link to sequential methods.) | |
| Aug 23, 2021 at 21:04 | answer | added | Jeremy Miles | timeline score: 7 | |
| Aug 23, 2021 at 21:00 | history | tweeted | twitter.com/StackStats/status/1429911342291505155 | ||
| Aug 23, 2021 at 20:59 | answer | added | Michael Lew | timeline score: 7 | |
| Aug 23, 2021 at 20:47 | comment | added | Jeremy Miles | @Dave - they are the inverse if I look at whether the CIs include zero. I don't. | |
| Aug 23, 2021 at 20:46 | comment | added | Jeremy Miles | @Lewian - I don't think this is the same as sequential analysis, or has the same issues, because I'm not testing for significance or looking at where the upper or lower CI land. All I care about is the size of the CI. If it's significant early, I keep going. If it's not significant later, I stop. | |
| Aug 23, 2021 at 20:41 | comment | added | Dave | Keep in mind that hypothesis tests and confidence intervals are inverses of each other. | |
| Aug 23, 2021 at 20:35 | comment | added | Christian Hennig | If you do it straight as you describe it, it's indeed invalid. The keyword for what you're looking for is "Sequential Analysis" en.wikipedia.org/wiki/Sequential_analysis although I'm not an expert on that. | |
| Aug 23, 2021 at 18:15 | history | asked | Jeremy Miles | CC BY-SA 4.0 |