Is feature scaling beneficial to model fitting for all loss functions?

Question

When fitting a model — say, finding weights (coefficients) in a linear regression — the loss function is used to determine the quality of a fit. Let's say that the loss function for a set of weights $\boldsymbol{w}$ depends on the absolute difference between the model prediction $\hat{x}$ and observation $\boldsymbol{x}$ in feature space, e.g.,

$$ loss(\boldsymbol{w}) = \sum_{i=1}^N (\hat{x_i}(\boldsymbol{w})-x_i)^2 $$

where index $i$ indicates the $i^{th}$ feature in the model. In this case, a feature with a very large scale (say, its values range from -2000 to 2000) would be far more influential in the fit than a feature with a very small scale (say, its values range from -0.01 to 0.01). In this case, feature scaling — e.g., z-standardizing the values of all of our features — should help us converge on the right fit.

However, let's say that we're looking at a classifier with a loss function that depends upon the feature vector $\boldsymbol{x}$ only through:

$$ \sigma(\boldsymbol{w}\cdot\boldsymbol{x}) \equiv \frac{1}{1+\exp(-\boldsymbol{w}\cdot \boldsymbol{x})} $$

where $w$ is a set of weights. This might occur, for example, when looking at the LogLoss for a binary classifier, in which for the $t^{th}$ observation and labels $y_t \in \{0,1\}$:

$$ loss(\boldsymbol{w}_{t}) = -y_t \log p_t - (1-y_t )\log (1-p_t) $$

and $p_t = \sigma(\boldsymbol{w}\cdot \boldsymbol{x})$. In this case, it appears that the weights can 'absorb' the scaling of individual features without an issue, preventing that scale from affecting the fit.

Question: Am I correct in thinking that in such cases, feature scaling is not necessary in order to achieve a good fit? Are there other features of the model that might suffer if we fail to scale our features?

Answer 1

This latter case is very common in, for example, logistic regression. In this case, you are correct that the model is the same in either case. It also shows up in linear regression, where the usual loss is

$$ \mathit{loss}(w) = \lVert y - (X w + b) \rVert^2 $$ (with $y \in \mathbb R^N$ the vector of labels, $X \in \mathbb R^{N \times d}$ the feature matrix, $w \in \mathbb R^d$ the weights and $b \in \mathbb R$ the offset).

Scaling still matters, though:

• If you're regularizing the loss, e.g. in ridge regression $$ \mathit{loss}(w) = \lVert y - (X w + b) \rVert^2 + \lambda \lVert w \rVert^2 .$$ Here if you change the relative scales of $X$, equally rescaling $w$ will change your loss by changing the $\lVert w \rVert^2$ term.

• If you're solving with iterative methods like gradient descent, poor feature scaling leads to poor conditioning of the problem, which can make your gradient descent much slower.

• In interpretation of the model weights $w$. (If your data is standardized, it's fairly reasonable to interpret the features with the largest model weights as the most important; you certainly can't do that if the scales of the features vary widely.)