I have encountered some applications of the softmax (multinomial logistic regression) in neural network applications where the sum-to-one constraint is ignored (e.g. see this link or this link). That is, the second final layer, say $\vec u \in \mathbb R^l$ for some $l$, is mapped through some parametrized function $\vec \theta(\vec u)$ to yield the probability of each category:

$$P(Y = k\mid \vec u) = \frac{\exp\theta_k(\vec u)}{\sum_{i = 1}^K\exp\theta_i(\vec u)}$$

Because the probabilities sum to one we can simply set:

$$ \exp\theta_1(\vec u) = 1 $$

This yields the sigmoid (logit link) function as a special case as illustrated in this answer, here, and here. It, however, seems as if this is not done in the link and in a number of applications. That is, no restrictions are placed in $\vec\theta$. I have even seen softmax been used in the binary case (that is without using the logit link) which in the worst case results in twice the number of parameters.

My questions is whether there are some general arguments to be made for or against this. Not restricting $\vec\theta$ can:

  1. yield an infinite number solutions irrespectively of the network being used. Some of the solutions are even with parameters tending towards $\pm \infty$.
  2. yield a large number of additional parameters at best, as I see it, only increasing the cost of evaluating the loss and the gradient.

I would suspect that:

  • Convergence can be slower (more loss function and gradient evaluations are required). This at-least seems to be true in some small simulations I have done with a logistic regression. As stated here, the loss as a function of $\vec\theta$ is still convex but the Hessian is singular which can cause problems with the Newton method or quasi Newton methods.
  • Estimation may continue forever though this is presumably unlikely.

Anything that I am missing? Are there any papers to justify or argue against not restricting $\vec\theta$?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.