Are ridge and LASSO shrinkage parameters functions of the number of features? I was wondering if Ridge and LASSO shrinkage parameters (I am referring to the lambda in the canonical loss function) are functions of the number of features if one is targeting a "constant" shrinkage level?
An example is: let's suppose I determine my optimal shrinkage parameter to be some number x for a set of 4 features that are appropriately standardized. If I then bring a 5th feature into my feature set (also appropriately standardized), would I need to revise my previously selected shrinkage parameter if I wanted to target the same level of total shrinkage as before? Does the answer differ for Ridge vs. LASSO?
 A: To ground the discussion I will consider a problem with $n$ points and $p$ parameters. Furthermore I will assume that the data matrix $\boldsymbol{X}\in\mathbb{R}^{n\times{p}}$ has been whitened, such that $\boldsymbol{X}^T\boldsymbol{X}=n\boldsymbol{I}_{p\times{p}}$.
Then the question relates to a regularized M-estimation problem with objective function of type
$$E[\boldsymbol{c}]=\tfrac{1}{k}\|\boldsymbol{Xc}-\boldsymbol{y}\|_k^k+\tfrac{\lambda}{j}\|\boldsymbol{c}\|_j^j$$
for some regularization strength $\lambda\geq{0}$ and data/penalty norms $k,j\in\mathbb{N}$, and seeks a solution $\boldsymbol{c}\in\mathbb{R}^p$.
In the case of ridge regression ($k=j=2$) we have the solution
$$(n+\lambda)\boldsymbol{c}=\boldsymbol{X}^T\boldsymbol{y}$$
From this, we can see that the magnitude of a coefficient $c_i$ will be impacted by $\lambda/n$, the relative magnitude of the penalty vs. the number of data points.
In terms of $p$, the answer depends on what "shrinkage level" means. From the above solution, the magnitude of $|c_i|$ is not impacted by $p$, but the norm $\|\boldsymbol{c}\|$ may scale with $p$.
However commonly "shrinkage" is not really of direct interest per se. Instead,  the goal of the regularization is to prevent overfitting, in which case  generalization error is more relevant. In this context, it would be more appropriate to set the value of $\lambda$ based on something like cross-validation. Then the general expectation would be that the optimal $\lambda$ would tend to increase with $p$.
I believe these same considerations should apply to the general problem, regardless of the values of $k$ and $j$ (i.e. also for LASSO, where $k=2$, $j=1$).
