I am trying to optimize sklearn.linear_model.SGDRegressor, and I was wondering if people could point me in which direction I should try to optimize? Personal experience and literature are both welcome.
I think it would be much better if there is kind of a ranking in which way to search; should I start with finding a good alpha, or with a learning rate? Do you also have to do back checks at some point? What is the common way of searching this specific (or other sklearn/machine learning related) multivariate space? How dependent are these parameters on each other?
I understand this matters per situation, but how to do this kind of optimizing in a smarter way than level wise search?
I'm also never so sure when changing anything could have an effect on these paramaters. Basically if I would take a smaller sample, then I have no idea how solid the "optimal" values I found before, are.
E.g. some possible parameters to tune:
loss : ‘squared_loss’, ‘huber’, ‘epsilon_insensitive’, or ‘squared_epsilon_insensitive’ alpha : float l1_ratio: float between 0 (L2) and 1 (L1) epsilon: float learning_rate : optional eta0 : double, optional power_t : double, optional