I am trying to optimize sklearn.linear_model.SGDRegressor, and I was wondering if people could point me in which direction I should try to optimize? Personal experience and literature are both welcome.

I think it would be much better if there is kind of a ranking in which way to search; should I start with finding a good alpha, or with a learning rate? Do you also have to do back checks at some point? What is the common way of searching this specific (or other sklearn/machine learning related) multivariate space? How dependent are these parameters on each other?

I understand this matters per situation, but how to do this kind of optimizing in a smarter way than level wise search?

I'm also never so sure when changing anything could have an effect on these paramaters. Basically if I would take a smaller sample, then I have no idea how solid the "optimal" values I found before, are.

E.g. some possible parameters to tune:

loss : ‘squared_loss’, ‘huber’, ‘epsilon_insensitive’, or        
alpha : float
l1_ratio: float between 0 (L2) and 1 (L1)
epsilon: float
learning_rate : optional
eta0 : double, optional
power_t : double, optional

2 Answers 2


Leon Bottou's webpage has some good references on how to use SGD wisely(here is a link that you may find useful).


There's a few options.

One is to search these hyper-parameters randomly then pick the one with the best cross-validation. I tend to run small experiments, record my results and then narrow the parameters. RandomizedSearchCV is a good starting point.

After getting a decent estimate of good hyperparameters, with some modules, you can set the pass the argument, verbosity=9, then look at the output at each iteration. Learning_rate, epsilon, eta0, n_iters, and power_t are influenced by one another. It's even more important for tuning Gradient Boosting. Personally I like this approach because I've been too lazy to throw down some code for the next paragraph. I also like the fast iteration associated with it. I can tune a model close to real-time, and stay in the feedback loop.

If you're willing to wade in undocumented water, then you can try to optimize over the hyperparameters with a Tree of Parzen Estimators, which has been shown in some cases to give a performance boost over Gaussian Process Confidence Bound methods but not in all. These are probably a much better approach, but their evaluation is time-consuming.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.