0
$\begingroup$

What should be a better initialization for weights of ridge regression if I have to perform gradient descent. I have tried with all weights 0, all weights 1, and random initialization. In all the cases, random worked better. Is this true that random works better than others generally for ridge regression or is there a better way to initialize the weights so that the gradient descent algorithm converges faster?

$\endgroup$
0
$\begingroup$

A fixed specification of the initial value can lead to a far distance to the optimal value if mis-specified (which you do not know beforehand, otherwise you would know the solution).

Hence, in the absence of any strong prior belief about optimal parameter regions and given that you are using gradient-descent type optmization routines it is always better to seed your search with random choices (with a sensible choice of the initial parameter value distribution). But ensure that you are doing this for several runs, not just one. Bayesian people like to do that and it is in many cases a successful approach.

Note, however, If you have strong domain knowledge and have solved similar problems before, it can be more efficient to set the initial values yourself given your experience instead of a non-informative (random) prior.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.