I'm running a Ridge regression with about 1200 parameters (and about 30000 datapoints). I noticed that for some values of ridge, the weights look qualitatively different beyond a certain point.
In particular, higher values for the ridge parameter will yield weights that clearly have some kind of structure to them, whereas at some point they look almost like white noise. In this case, I'd expect the weights to have the former kind of structure due to the features I'm using, but cross-validation often chooses a parameter for which the weights look totally noisy.
It seems to me that the ridge parameter should only change the relative distribution of the weights, but not their qualitative differences. Am I wrong here?