I have a problem. I am a biologist working in machine learning. Recently I am dealing with LASSO regularized linear regression. Very nice RMSE, MSE, R^2 values under all kinds of cross-validation. But the problem is that biologists at the moment (especially here in Eastern Europe) are not willing to let go of this p value thing. My supervisor wants me to show some kind of p-value... Do you think it makes sense to do a Pearson correlation analysis on the test set between predicted and actual values and show the p-value derived from it? What exactly would that tell me? I refrain from doing this, but I'll have to do something...
3
-
$\begingroup$ Welcome to Cross Validated! What hypothesis would you want to test? Can you form a null and alternative hypothesis? Just the null? Just the alternative? $\endgroup$– DaveCommented Feb 26 at 19:04
-
1$\begingroup$ stats.stackexchange.com/questions/291409/… $\endgroup$– John MaddenCommented Feb 26 at 19:11
-
$\begingroup$ This page has some discussion about this issue, with links to references and an R package that might be helpful. $\endgroup$– EdMCommented Feb 26 at 19:41
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It sounds like your supervisor wants to show p-values for coefficients similar to those that you would show for unpenalized regression. You can't just plug the selected predictors into an unpenalized regression model and get their p-values, as that ignores the fact that the data were used in predictor selection.
There is a way, however, to take that into account, as explained in Section 6.3.2.1 of Statistical Learning with Sparsity (SLS). For any fixed value of the penalty, with that method you "can obtain exact p-values and confidence intervals for the parameters of the active set in the lasso solution...conditional on the membership and signs of the active set" (SLS, pp. 154-155). You can think of that as correcting for the use of the data to have come up with this particular set of predictors. It "tests whether the coefficient of any given predictor is zero in the projected model" (Section 6.3.3). The R selectiveInference package implements that and some related methods.
One potential problem is that you presumably used cross-validation to choose the penalty value, rather than pre-specifying a penalty. SLS claims: "Simulations suggest this does not widen the confidence intervals substantially" (page 155). So that might still be the best way to proceed in your situation.
All of the above, however, ignores the problem of the probability of a particular predictor being included in the final set of predictors. If you have multiple correlated predictors (as is often the case with predictors like gene-expression values), the choice of which to include in the final model can be very data-set dependent. For your own benefit, you might assess the variability of predictor selection by repeating all steps of the modeling on multiple bootstrap samples of the data, as illustrated in Section 6.2 of SLS. That variability in selected predictors doesn't necessarily affect the predictive value of the model (which, as you understand, is what really matters), but it does tend to emphasize LASSO's somewhat arbitrary choice of "the" predictor set.
