5
$\begingroup$

I have a regularized linear regression model with a large number of parameters (~100,000) that has been optimized using a sparse fitting algorithm (coordinate descent with early stopping). The model parameters and prediction accuracy were cross-validated using repeated random subsampling. I want to assess the statistical significance of individual parameters using the t-ratio (mean coefficient divided by standard deviation of the resampled estimates). Is correction for multiple comparisons appropriate in this case and if so is it appropriate to select only the non-zero weights for significance testing? Correction for multiple comparisons of individual regression parameters does not seem to be a common practice (e.g., correcting for testing significance of multiple main effects in ANOVA).

$\endgroup$

2 Answers 2

4
$\begingroup$

In the context of model selection, hypothesis tests can be used either for the (a) selection of the model, or (b) the verification of selected predictors. Case (a) includes forward/backward/step-wise procedures, which are very common. These procedures however, do not try to account for the massive amount of hypotheses being tested along the way, so they do not really provide the type I error rate you might expect. A "model growth" approach that does try to control for the massive amount of hypotheses being tested, can be found (for instance) in Benjamini, Gavrilov- A simple forward selection procedure based on false discovery rate control.

In case (b), you have already selected a subset of predictors and then wish to verify if they are indeed significant. Note however, that these procedures, answer different questions; It is possible that you will get a smaller prediction error by omitting a significant predictor. It is also possible that adding a non significant predictor will improve your generalization error (if you have multicollinearity for instance). For these reasons, if you are only interested in prediction (and not in "explanation"), there is indeed no need for hypothesis testing. To the best of my knowledge (I might easily be wrong), properly controlling the type I error, after variable screening using cross-validation is an open question.

Concluding- if you decided you do need hypothesis testing for the purpose of interpretation, consider doing it from the start and not after a cross-validation screening. Otherwise, there are many heuristics, but no guarantee on the type I error rate.

$\endgroup$
2
$\begingroup$

I am not an expert in this area. But as far as I know, as mentioned in the previous answer, statistical inference in regularized models like LASSO is very much an open question. Relatively few papers deal with the inference part of regularized regression models (most are concerned with predictive properties). The paper P-Values for High-Dimensional Regression by Meinshausen et al. may be of interest to you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.