I would like to understand if we should remove variables with less p-Values after regression with regularization.
If p-value is high doesn't it mean that our beta is just by chance and it might actually be zero?
If so, then this beta should ideally be removed. Is there any case where I should be careful about this conclusion? When can p-Values after regularization be misleading and why?
I followed the answer here: https://stats.stackexchange.com/a/171462/172758
With even just 2 collinear predictors, their individual regression coefficients are likely to vary widely among bootstrap samples so that their individual p-values may appear insignificant. Nevertheless, their joint contributions to the regression might be much more stable and thus their combination very significant in practical terms
If at all two collinear variables' combination is more significant together wouldn't this be true even of p-Values of pure least squares regression? Or only regularization p-Values be misleading?