# How to find parameters in multivariate space efficiently?

I am trying to optimize sklearn.linear_model.SGDRegressor, and I was wondering if people could point me in which direction I should try to optimize? Personal experience and literature are both welcome.

I think it would be much better if there is kind of a ranking in which way to search; should I start with finding a good alpha, or with a learning rate? Do you also have to do back checks at some point? What is the common way of searching this specific (or other sklearn/machine learning related) multivariate space? How dependent are these parameters on each other?

I understand this matters per situation, but how to do this kind of optimizing in a smarter way than level wise search?

I'm also never so sure when changing anything could have an effect on these paramaters. Basically if I would take a smaller sample, then I have no idea how solid the "optimal" values I found before, are.

E.g. some possible parameters to tune:

loss : ‘squared_loss’, ‘huber’, ‘epsilon_insensitive’, or
‘squared_epsilon_insensitive’
alpha : float
l1_ratio: float between 0 (L2) and 1 (L1)
epsilon: float
learning_rate : optional
eta0 : double, optional
power_t : double, optional