# Is feature scaling beneficial to model fitting for all loss functions?

When fitting a model — say, finding weights (coefficients) in a linear regression — the loss function is used to determine the quality of a fit. Let's say that the loss function for a set of weights $\boldsymbol{w}$ depends on the absolute difference between the model prediction $\hat{x}$ and observation $\boldsymbol{x}$ in feature space, e.g.,

$$loss(\boldsymbol{w}) = \sum_{i=1}^N (\hat{x_i}(\boldsymbol{w})-x_i)^2$$

where index $i$ indicates the $i^{th}$ feature in the model. In this case, a feature with a very large scale (say, its values range from -2000 to 2000) would be far more influential in the fit than a feature with a very small scale (say, its values range from -0.01 to 0.01). In this case, feature scaling — e.g., z-standardizing the values of all of our features — should help us converge on the right fit.

However, let's say that we're looking at a classifier with a loss function that depends upon the feature vector $\boldsymbol{x}$ only through:

$$\sigma(\boldsymbol{w}\cdot\boldsymbol{x}) \equiv \frac{1}{1+\exp(-\boldsymbol{w}\cdot \boldsymbol{x})}$$

where $w$ is a set of weights. This might occur, for example, when looking at the LogLoss for a binary classifier, in which for the $t^{th}$ observation and labels $y_t \in \{0,1\}$:

$$loss(\boldsymbol{w}_{t}) = -y_t \log p_t - (1-y_t )\log (1-p_t)$$

and $p_t = \sigma(\boldsymbol{w}\cdot \boldsymbol{x})$. In this case, it appears that the weights can 'absorb' the scaling of individual features without an issue, preventing that scale from affecting the fit.

Question: Am I correct in thinking that in such cases, feature scaling is not necessary in order to achieve a good fit? Are there other features of the model that might suffer if we fail to scale our features?

$$\mathit{loss}(w) = \lVert y - (X w + b) \rVert^2$$ (with $y \in \mathbb R^N$ the vector of labels, $X \in \mathbb R^{N \times d}$ the feature matrix, $w \in \mathbb R^d$ the weights and $b \in \mathbb R$ the offset).
• If you're regularizing the loss, e.g. in ridge regression $$\mathit{loss}(w) = \lVert y - (X w + b) \rVert^2 + \lambda \lVert w \rVert^2 .$$ Here if you change the relative scales of $X$, equally rescaling $w$ will change your loss by changing the $\lVert w \rVert^2$ term.
• In interpretation of the model weights $w$. (If your data is standardized, it's fairly reasonable to interpret the features with the largest model weights as the most important; you certainly can't do that if the scales of the features vary widely.)