Computing a p-value using bootstrap I have a relatively complicated model-fitting procedure, the output of which is a set of coefficients $\beta$ (one for each feature in the training set) and a set of hyperparameters $\lambda$ (which control the amount of regularization).
I would like to compute p-values for each feature. At the moment, my procedure is this:


*

*Determine good values for the hyperparameters $\lambda=\lambda^*$ using cross-validation.

*Fit the model on the whole data set with these hyperparameters, obtaining $\beta=\beta^*$

*For each feature $i$

*

*For $j=1$ to $n$

*

*Shuffle the values of feature $i$ to produce a new data set $X^j$

*Fit the model to $X^j$ with the hyperparameters $\lambda$ to obtain $\beta^j$


*Count the proportion of times that $|\beta^j_i| > |\beta^*_i|$ and call this the p-value for feature $i$



This matches my intuitive understanding of what a p-value is (i.e. each of the reshufflings produces a data set where the null hypotheses $\beta_i=0$ holds, since I have randomized away all the dependence) but I wondered how 'canonical' it is. I can think of a couple of other ways to do it, for example:


*

*Refit the hyperparameters as well as the coefficients on each inner loop.

*Use sampling with replacement to generate the new data set, instead of shuffling.


Are either of those better?
And, more importantly, can the numbers I'm generating be interpreted as p-values with any degree of reliability?
 A: Great set of alternatives for generating a discussion!
1) Your primary suggestion is actually a form of permutation test rather than a bootstrap, and does indeed generate p-values.  The p-values are exact under the assumption of exchangeability, which I can't describe in terms specific to your model as I don't know your model, and if you can do all possible permutations.  In a linear regression, assuming the residuals are i.i.d. is sufficient for exchangeability to hold. However, it would be proper to refit the hyperparameters inside the loop.  
Note that in unbalanced ANOVA problems with missing cells it may not be possible to devise a permutation test because of the exchangeability requirement (Good, p. 138, for an example.)  
2) Using sampling with replacement turns the permutation test into a nonparametric bootstrap-based test.  Once again,it would be proper to refit the hyperparameters inside the loop, but exchangeability is not required.
Generally a nonparametric bootstrap test isn't exact or conservative, and it's less powerful than a permutation test, so I'd prefer the permutation test (if I have a choice.)  There are also parametric bootstraps which can be used when the distribution of the test statistic under the null hypothesis is known.  These are more powerful than their nonparametric cousins.  In your case, if you have a linear regression and you are willing to assume i.i.d. Normal residuals, the distribution of $\hat{\beta}$ under the null hypothesis is known and you can use that fact to construct a parametric bootstrap.  Still, you're not going to beat the permutation test.
A good reference for this is Permutation, Parametric, and Bootstrap Tests of Hypotheses (Good).
