# Softmax Regression Large Inner Product Float Overflow

In softmax regression, the probability $P$ that an item is part of class $l$ is given by $$P(y^{(i)}=l | x^{(i)};\theta)=\frac{e^{\theta^Tx^{(i)}}}{\sum_{j=1}^k e^{\theta^Tx^{(j)}}}$$

I have implemented softmax regression, but seem to come into problems when the inner product $\theta^Tx^{(i)}$ is large (say, greater than 30-50,) in which case even a 64 bit float overflows. What's the recommended procedure to avoid this problem? I've been thinking about normalizing the input vector $x^{(i)}$ such that $x^{(i)\prime}=\frac{x^{(i)}}{|x^{(i)}|}$, but that could make prediction less dynamic.

• How large is the dimension of $\theta$? Also, what is the range of entries in $\theta$ and $x$? – Sobi Dec 14 '15 at 3:48

You can subtract a constant from the $\theta^T x^{(i)}$ before taking the exponent. I believe that the max, per example, is commonly used.
Use logarithms! For example, instead of calculating $a/b$, try $exp(log(a) - log(b))$
• Step 1: consider the logarithm of the probabilities. I.e., $$\log \left(P(y^{(i)}=k | x^{(i)};\theta)\right)=\frac{\log \left(e^{\theta_k^Tx^{(i)}}\right)}{\log \left(\sum_{c=1}^K e^{\theta_c^Tx^{(i)}}\right)}$$
Example of how the log-sum-exp trick works in Naive Bayes explains those two steps in more details. Just replace underflows with overflows, and pretend $p(\mathbf{x}|Y=C_k)p(Y=C_k) = e^{\theta_c^Tx^{(i)}}$.