Correcting for multiple testing is easy, and there's a lot of literature around it. But now consider a different problem.

Say you have $X$ and $Y$ and you want to estimate $E[Y|X=x]= f(x)$ where $f$ could be any function.

Say you fit several functions $f_1, \dots, f_k \in F$ and you pick one of them according to some criteria (for example, via cross-validation or whatever other criteria you prefer). Let $f^*$ be the chosen function.

Now you want to do inference with $f^*$. For instance, suppose $f^*$ has some parameteric form (for example, a GLM) and you want to obtain confidence intervals for some of the parameters.

If you had started with $f^*$ to begin with, then inference is easy, just traditional statistical techniques. But you didn't start with $f^*$. So how can you do inference properly, taking into account the fact you have searched $f^*$ over a set of functions $F$?


I would dispute that "correcting for multiple testing is easy"...

That said, if you have a clear algorithm for selecting $f^\ast$, you could use simulation or permutation to obtain $p$-values that incorporate this selection. Simulate data under the null hypothesis for whatever effect you are interested in, then run the entire process of both selecting $f^\ast$ and performing inference on it, saving the resulting statistic. Do this many times with randomly generated data. Finally insert your actually observed statistic into this null distribution of statistics, and you have a $p$-value that takes the selection into account.

| cite | improve this answer | |
  • $\begingroup$ This makes sense... but simulate data or bootstrap data? Also, can we prove this gives us what we want? I don't want a p-value by the way, but a confidence interval with valid coverage. $\endgroup$ – statslearner Jan 25 '18 at 2:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.