Random model selection and validity of significance tests Suppose we have some data, $\{y_i, x_{1i}, \dots, x_{ki}\}_{i=1}^n$ and we want to build a linear model of the form $y_i = \beta_0 + \beta_{1'i}x_{1'i} + \dots + \beta_{k'i}x_{k'i} + \epsilon_i$, where the primed indexes are a subset of $\{1,\dots, k\}$. There are $2^k$ possible models to choose from. Now if we randomly (uniformly) draw a model from all $2^k$ , and fit it to the data, once, can we still interpret the standard errors and CIs as usual?
 A: The meaning of standard errors and CIs is predicated on the model being correct. 
Even assuming that both the class of models is correct and the set of candidate variables is complete, nevertheless a lot of those models will exclude some of the necessary variables, and so even their parameter estimates will be biased (also see here, especially the diagram).
As a result, the problem is more basic than one of standard errors and width of confidence intervals.
What a randomly selected model avoids is the bias introduced by variable selection - bias in both standard errors and estimates, but at the expense of omitted-variable bias, so it's not necessarily preferable.
Typical advice leans toward using hold-out samples (such as via cross-validation). If selection and inference are assessed on different subsets of the data, the effect of variable selection bias on inference can be avoided.
If that's not possible for some reason, fitting all $k$, but regularizing in some way (such as via shrinkage) would partly mitigate some of the earlier problems.
