0
$\begingroup$

In a machine learning algorithm that I'm using, I need to get the exponential values of something in one of the steps.

This is the step that I'm dealing with right now: enter image description here

I've already got all the 1+g_j(X_i) etc etc calculated, there's no problem in that. Let's call it calculated_value. That's fine.

The problem is, I am getting infinities and overflows when implementing this function in code.

What's the solution for this? What would the logarithmic version of the same function, that I could use in place of the function above?

In case it matters, β is going to be used as a coefficient to some variables, including partially to calculate the weights for a weighted regression solution.

$\endgroup$
1
  • $\begingroup$ A very good approximation for $\ln(1+\exp(x))$ for $x>7$ or so is $x+\exp(-x)$. This means you can readily take logs of numerator and denominator and work on log-scales when dealing with large arguments. $\endgroup$
    – Glen_b
    May 8, 2014 at 3:46

1 Answer 1

1
$\begingroup$

One way to handle this would be to rescale the terms in the numerator and denominator by a suitably large constant $C$, which is equivalent to subtracting $log(C)$ from the numerator of each exponential terms as follows. I'll reduce the notation of the problem for simplicity.

$\beta_j(X_i) = \frac{e^A}{1 + e^A + e^B}= \frac{ C^{-1}e^A}{C^{-1} + C^{-1}e^A + C^{-1}e^B} = \frac{ e^{A-log(C)}}{C^{-1} + e^{A-log(C)} + e^{B-log(C)}}$

One possible choice would be to let $log(C) = max(A,B)$. Then at least you are guaranteed to not have overflow!

$\endgroup$
3
  • $\begingroup$ Is the formula that you're proposing a kind of "normalization"? (I'm still learning here..) $\endgroup$
    – user961627
    May 8, 2014 at 10:51
  • $\begingroup$ In fact, even the function I asked about above, that too is a kind of normalization, right? $\endgroup$
    – user961627
    May 8, 2014 at 11:30
  • 1
    $\begingroup$ Sure, you could call it that. The function you asked about is itself a normalization, but the normalization fails if any one of the exp(x) is inf! $\endgroup$
    – jsk
    May 8, 2014 at 15:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.