# Batch loss of objective function contains exp becomes nan

I am trying to solve a survival analysis problem, where all data are either left-censoring or right-censoring. I use an objective function which contains the CDF of Gumbel distribution.

I have $m$ features and $m+1$ coefficients which need to be learned. The scale of the distribution, $\lambda$ can be represented by a linear regression. Since the scale must be a positive number, I use softplus. (I think an exp transformation may be easy to go unlimited.)

$\lambda = softplus(\theta_0 + \sum_{j=1}^{m} \theta_jx_j ) = ln[ 1+exp(\theta_0 + \sum_{j=1}^{m} \theta_jx_j) ]$

The scale is fed into a Gumbel distribution.

$h(t) = \int_{0}^{time}e^{-e^{-(t-\mu)/\lambda}}dt$, where the location $\mu$ is pre-specified.

$h(t)$ is the probability that the patient is dead before time $t$, i.e., left-censoring. $1 - h(t)$ is the probability that the patient is dead after $t$, i.e., right-censoring.

In the ground truth, binary target, $y^{(i)}$, is whether the patient is left-censoring. As the model outputs how likely the patient is left censoring at $t$, I use log-loss to measure the loss of the model.

I use Tensorflow to implement the model:

    input_vectors = tf.placeholder(tf.float32,
shape=[None, num_features],
name='input_vectors')

time = tf.placeholder(tf.float32, shape=[None], name='time')
event = tf.placeholder(tf.int32, shape=[None], name='event')

weights = tf.Variable(tf.truncated_normal(shape=(num_features, 1), mean=0.0, stddev=0.02))
scale = tf.nn.softplus(self.regression(input_vectors, weights))
'''
if event == 0, right-censoring
if event == 1, left-censoring
'''
not_survival_proba = self.distribution.left_censoring(time, scale)  # the left area
logloss = tf.losses.log_loss(labels=event, predictions=not_survival_proba)


The implementation of the Gumbel distribution:

class GumbelDistribution:
def __init__(self, shape=0.01):
self.shape = shape  # this param is actually called "location" in Statistics

def left_censoring(self, time, scale):
return tf.exp(-1 * tf.exp((self.shape - time) / scale)) - tf.exp(-1 * tf.exp(self.shape / scale))

def right_censoring(self, time, scale):
return 1 - tf.exp(-1 * tf.exp((self.shape - time) / scale))


However, the batch loss becomes NaN after several iteration. After I change the distribution to Weibull. It works. So I guess the problem is the two $exp$s in the CDF of Gumbel.

Epoch 1 - Batch 1/99693: batch loss = 16.3606
Epoch 1 - Batch 2/99693: batch loss = 25.5445
Epoch 1 - Batch 3/99693: batch loss = 17.1181
Epoch 1 - Batch 4/99693: batch loss = 10.6815
Epoch 1 - Batch 5/99693: batch loss = 17.2127
Epoch 1 - Batch 6/99693: batch loss = 28.7549
Epoch 1 - Batch 7/99693: batch loss = 13.8332
Epoch 1 - Batch 8/99693: batch loss = 19.3377
Epoch 1 - Batch 9/99693: batch loss = 19.7385
Epoch 1 - Batch 10/99693: batch loss = 17.7479
Epoch 1 - Batch 11/99693: batch loss = 13.1403
Epoch 1 - Batch 12/99693: batch loss = 15.0979
Epoch 1 - Batch 13/99693: batch loss = 17.5434
Epoch 1 - Batch 14/99693: batch loss = 21.5072
Epoch 1 - Batch 15/99693: batch loss = 10.4660
Epoch 1 - Batch 16/99693: batch loss = 26.9554
Epoch 1 - Batch 17/99693: batch loss = nan
Epoch 1 - Batch 18/99693: batch loss = nan
Epoch 1 - Batch 19/99693: batch loss = nan
Epoch 1 - Batch 20/99693: batch loss = nan
Epoch 1 - Batch 21/99693: batch loss = nan
Epoch 1 - Batch 22/99693: batch loss = nan
Epoch 1 - Batch 23/99693: batch loss = nan


Any idea how to solve this problem?

• There's nothing inherently problematic about two exponentials. The problem occurs in whatever algorithm you are iterating, but since you provide no details, there's no information available to answer your question.
– whuber
Commented Aug 6, 2018 at 16:27
• @whuber I have added more details. Please unhold it. Thanks. Commented Aug 6, 2018 at 20:29

Multiple exponentials are generally a recipe for disaster. If you truly believe your output data $Y$ is standard Gumbal (after scaling), then $\exp(-Y)$ is exponentially distributed. This should get rid of one exponential in your training. The next step is to investigate $\lambda$. You seem to allow for $\lambda=0$, which will squash your exponential. Consider using a small hyperparameter $\epsilon$ in your softplus, i.e. $\ln(1+\epsilon+\exp(\cdots))$. Also it looks like $t-\mu$ can be positive or negative? This coupled with a tiny $\lambda$ will cause blowup.

Next, you've chosen weights to have mean 0 and standard deviation $0.02$. Double check that this produces sensible scales for your softplus and your $h(t)$, by predicting on a sample of your $x$'s just before running the optimization. Are your inputs $x$ normalized?

Finally, your loss function looks something like $-y_i\ln(h(x_i))-(1-y_i)\ln(h(x_i))$. You can remove numerical headaches here by manually simplifying this. If $h$ is Gumbal, then the first logorithm dissapears. If $h$ is exponential (after the above transformation), then the first logorithm will dissapear, and the second logorithm will act on a softmax (when properly rewritten). Use tensorflow's native softmax implementation for the second one, to ensure you're not blowing up.

• For the first suggestion, do you mean I can transform y to -ln(y) so as to remove one exp? Commented Aug 12, 2018 at 1:55
• Yes. It's easier to see in reverse. If $X$ is exponentially distributed, then $P(X\leq x)=1-e^{-\lambda x}$, so $P(e^{-X}\geq e^{-x})=1-e^{-\lambda e^{-x}}$, i.e. $e^{-X}$ is Gumbal. Commented Aug 12, 2018 at 22:46
• So my model uses Exp distribution (instead of Gumbel) and outputs $-ln(\hat y)$. Just wanna confirm, in this case, the loss cannot be measured by LogLoss, since $-ln(y)$ is not a probability. Maybe RMSD or anything more similar to Logloss? Commented Aug 13, 2018 at 21:23

This is probably due to one or more of the following: