1
$\begingroup$

In scikit-learn the Ridge regression estimator has a normalize parameter that normalizes the regressors. I found that it was necessary to set this to True to get a reasonable fit to my data when using higher degree polynomial features (it provided consistent regularization no matter how many samples I trained/predicted on)

I would like to use a more robust estimator such as Huber regression, however it does not have this normalize parameter so the fit is quite poor.

Sklearn has a preprocessing.Normalizer() transformer which I tried adding into my pipeline, but it didn't help. When I instead created a preprocessing.FunctionTransformer() that calls preprocessing.normalize(), I found that if I set axis=0 (i.e. normalizing over the features rather than samples) I got a good fit much like when I had set normalize=True for the Ridge estimator.

However, this only worked when I predicted on a sample of similar size to my training set. Depending on the number of inputs, the predicted values would change (this behavior does not occur with Ridge's normalize=True)

I've been reading through the Ridge estimator's source code trying to find exactly how it implements its normalize parameter, but it seems like a very convoluted solution.

Is there a relatively straightforward way that will properly normalize the regressors of a Huber estimator in the same way that the normalize parameter does for the Ridge estimator?

$\endgroup$
  • $\begingroup$ There are two useful Huber algorithms: Huber Scaling and Huber Regression. They are available in statsmodels package. It would be nice to see those both covered in sklearn. The scaling might be useful for the normalisation. $\endgroup$ – Nick Clarke Jan 30 at 15:22
2
$\begingroup$

IIUC, it seems like you've confused two different forms of normalization.

sklearn.preprocessing.Normalizer normalizes vectors to unit norm. Note how it is naturally used to scale rows (instances), and not columns (features). Unit normalization is dependent on the vector length, in general. Concatenate a vector to itself, and you will need to reduce the elements further in order to retain unit length. For rows (instances), the length is constant.

sklearn.preprocessing.StandardScaler, conversely, removes the mean, and scales to unit variance. This is naturally used to scale columns (features). It is basically independent of the vector size.

In your case, it seems like you should use StandardScaler together with something like sklearn.linear_model.SGDRegressor with (Huber loss) in a pipeline. You will need somehow to tune the l1 and and l2 parameters, preferably using some form of cross validation.

$\endgroup$
  • $\begingroup$ Switching from Normalizer to StandardScaler helped. It still irks me that it doesn't completely replicate the behavior of setting normalize=True (testing both methods on the Ridge estimator I got slightly different predictions). I've been using sklearn.linear_model.HuberRegressor for Huber loss, this note in the sklearn docs makes it sound more appealing than SGDRegressor. It also sounds like StandardScaler is sensitive to outliers so I'll have to see if there's a robust equivalent of that or make my own $\endgroup$ – mrossman Jul 26 '17 at 20:21
  • $\begingroup$ @mrossman You might find interest in RobustScaler. As to the normalize parameter, it shouldn't be too difficult to wrap the combination of a scaler + HuberRegressor into one step.` $\endgroup$ – Ami Tavory Jul 26 '17 at 20:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.