# Eliminating redundancy in Multinomial Logistic model

At page 348 - Chapter 10.6 in The Elements of Statistical Learning (12th printing -2007), the logistic model for K-class classification is expressed as: $$p_k(x)={exp(f_k(x))\over \sum_{l=1}^{K}exp(f_l(x))}$$. Then, the book states that "There is a redundancy in the functions $$f_k(x)$$, since adding an arbitrary $$h(x)$$ to each leaves the model unchanged". Thus the constraint $$\sum_{k=1}^{K}f_l(x) = 0$$ is imposed to eliminate the redundancy.

Why does imposing this constraint help to eliminate the redundancy?

It is because you only need to provide a linear combination of the inputs, $$f_k(x)$$, for all but one of the classes, and the $$f_k(x)$$ for the remaining class can be inferred from the constraint (as they all have to sum to zero). This is a very worthwhile saving if you are fitting the model using IRWLS as it involves inverting the Hessian matrix, which is $$\mathcal{O}(d^3K^3)$$, where $$d$$ is the number of inputs, and it effectively reduces $$K$$ by one.
• Thanks for your answer. My concern is if we impose this constraint, does it have any effect on the multinomial model? Given that $\sum p_k(x) = 1$, hence, there has been already a constraint on the $\{f_k(x)\}_1 ^ K$ Jan 2, 2022 at 17:28
• No, I don't believe so, at least unless the model is regularised. The sum of probabilities being one does not apply a constraint on the $\{f_k(x)\}_1^K$. If you add a h(x) to all of the $f_k(x)$, then that is equal to multiplying the $\exp\{f_k(x)\}$ by $\exp\{h(x)\}$, which cancels when you perform the division in the softmax function, so the trick is just exploiting that second redundancy. Jan 2, 2022 at 17:38
• Then, is the constraint $\sum k_1 ^ K f_k(x)$ play a role as $h(x)$? Jan 2, 2022 at 17:59
• As you can add an arbitrary h(x), then you can make the sum $\sum_{k=1}^Kf_k(x)$ be anything you like, so you may as well make it zero, in which case $f_K(x) = \sum_{k=1}^{K-1}f_k(x)$ Jan 2, 2022 at 18:05