Suppose I have variables $X_j$, $j=1,\ldots,p$, some of which are correlated, and some continuous output $y$.
I want to rank the variables by importance. One way is to do an association test of each variable j with the output (linear regression for each variable). This ignores the correlations between the variables, and potentially leads to selection of redundant variables in terms of prediction.
Another way is to build a multivariable model of all variables (potentially with some penalisation, l1 or l2, not material here). The multivariable model implicitly accounts for the correlations between the variables, so if two variables are correlated with the output and also high correlated with each other, usually the one more correlated with the output will get a higher regression coefficient, and the weaker one will get a low coefficient (or zero in case of lasso). This is contrast with the univariable approach where both will get a high coefficient.
Another way to say this is that the multivariable model includes all the potential confounders of the association of each variable j with the output but the univariable models is confounded most of the time unless it happens to look at the causal input.
Therefore, I would expect the estimates from the univariable methods to be inflated relative to the true effects, with a high false positive rate of detecting the causal variables.
Is anyone aware of literature discussing these intuitions more rigorously?
