I'm running a Ridge regression with about 1200 parameters (and about 30000 datapoints). I noticed that for some values of ridge, the weights look qualitatively different beyond a certain point.

In particular, higher values for the ridge parameter will yield weights that clearly have some kind of structure to them, whereas at some point they look almost like white noise. In this case, I'd expect the weights to have the former kind of structure due to the features I'm using, but cross-validation often chooses a parameter for which the weights look totally noisy.

It seems to me that the ridge parameter should only change the relative distribution of the weights, but not their qualitative differences. Am I wrong here?

  • $\begingroup$ With ridge parameter low, optimization problem becomes ill-defined, so it's not surprising that weights look messy. What is surprising is that CV chooses low ridge parameter value. Are you sure observations are independent? $\endgroup$ – AlexGenkin Jul 12 '14 at 16:59
  • $\begingroup$ No, they are not quite independent. In my case observations are actually moments in time from a recording of sound, which has been transformed into a spectrogram. I've downsampled this to 10ms bins, which are less correlated than originally, but certainly do still have autocorrelations in them. Do you have a reference for how this would affect the regression results? When I've tried to run the same regression using stochastic gradient descent with early stopping, the weights look reasonable once more... $\endgroup$ – choldgraf Jul 12 '14 at 19:02
  • $\begingroup$ With correlated observations CV no longer has strict sense, that's what I meant. $\endgroup$ – AlexGenkin Jul 14 '14 at 1:55
  • $\begingroup$ Ah - that does make sense. They're weakly correlated, but that might be making a difference. I'll try further downsampling to see if I can remove these effects. $\endgroup$ – choldgraf Jul 14 '14 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.