I am trying to solve a survival analysis problem, where all data are either left-censoring or right-censoring. I use an objective function which contains the CDF of Gumbel distribution.

I have $m$ features and $m+1$ coefficients which need to be learned. The scale of the distribution, $\lambda$ can be represented by a linear regression. Since the scale must be a positive number, I use softplus. (I think an exp transformation may be easy to go unlimited.)

$\lambda = softplus(\theta_0 + \sum_{j=1}^{m} \theta_jx_j ) = ln[ 1+exp(\theta_0 + \sum_{j=1}^{m} \theta_jx_j) ]$

The scale is fed into a Gumbel distribution.

$h(t) = \int_{0}^{time}e^{-e^{-(t-\mu)/\lambda}}dt$, where the location $\mu$ is pre-specified.

$h(t)$ is the probability that the patient is dead before time $t$, i.e., left-censoring. $1 - h(t)$ is the probability that the patient is dead after $t$, i.e., right-censoring.

In the ground truth, binary target, $y^{(i)}$, is whether the patient is left-censoring. As the model outputs how likely the patient is left censoring at $t$, I use log-loss to measure the loss of the model.

I use Tensorflow to implement the model:

    input_vectors = tf.placeholder(tf.float32,
                                  shape=[None, num_features],

    time = tf.placeholder(tf.float32, shape=[None], name='time')
    event = tf.placeholder(tf.int32, shape=[None], name='event')

    weights = tf.Variable(tf.truncated_normal(shape=(num_features, 1), mean=0.0, stddev=0.02))
    scale = tf.nn.softplus(self.regression(input_vectors, weights))
    if event == 0, right-censoring
    if event == 1, left-censoring 
    not_survival_proba = self.distribution.left_censoring(time, scale)  # the left area
logloss = tf.losses.log_loss(labels=event, predictions=not_survival_proba)

The implementation of the Gumbel distribution:

class GumbelDistribution:
def __init__(self, shape=0.01):
    self.shape = shape  # this param is actually called "location" in Statistics

def left_censoring(self, time, scale):
    return tf.exp(-1 * tf.exp((self.shape - time) / scale)) - tf.exp(-1 * tf.exp(self.shape / scale))

def right_censoring(self, time, scale):
    return 1 - tf.exp(-1 * tf.exp((self.shape - time) / scale))

However, the batch loss becomes NaN after several iteration. After I change the distribution to Weibull. It works. So I guess the problem is the two $exp$s in the CDF of Gumbel.

Epoch 1 - Batch 1/99693: batch loss = 16.3606
Epoch 1 - Batch 2/99693: batch loss = 25.5445
Epoch 1 - Batch 3/99693: batch loss = 17.1181
Epoch 1 - Batch 4/99693: batch loss = 10.6815
Epoch 1 - Batch 5/99693: batch loss = 17.2127
Epoch 1 - Batch 6/99693: batch loss = 28.7549
Epoch 1 - Batch 7/99693: batch loss = 13.8332
Epoch 1 - Batch 8/99693: batch loss = 19.3377
Epoch 1 - Batch 9/99693: batch loss = 19.7385
Epoch 1 - Batch 10/99693: batch loss = 17.7479
Epoch 1 - Batch 11/99693: batch loss = 13.1403
Epoch 1 - Batch 12/99693: batch loss = 15.0979
Epoch 1 - Batch 13/99693: batch loss = 17.5434
Epoch 1 - Batch 14/99693: batch loss = 21.5072
Epoch 1 - Batch 15/99693: batch loss = 10.4660
Epoch 1 - Batch 16/99693: batch loss = 26.9554
Epoch 1 - Batch 17/99693: batch loss = nan
Epoch 1 - Batch 18/99693: batch loss = nan
Epoch 1 - Batch 19/99693: batch loss = nan
Epoch 1 - Batch 20/99693: batch loss = nan
Epoch 1 - Batch 21/99693: batch loss = nan
Epoch 1 - Batch 22/99693: batch loss = nan
Epoch 1 - Batch 23/99693: batch loss = nan

Any idea how to solve this problem?

  • $\begingroup$ There's nothing inherently problematic about two exponentials. The problem occurs in whatever algorithm you are iterating, but since you provide no details, there's no information available to answer your question. $\endgroup$
    – whuber
    Commented Aug 6, 2018 at 16:27
  • 1
    $\begingroup$ @whuber I have added more details. Please unhold it. Thanks. $\endgroup$
    – Munichong
    Commented Aug 6, 2018 at 20:29

2 Answers 2


Multiple exponentials are generally a recipe for disaster. If you truly believe your output data $Y$ is standard Gumbal (after scaling), then $\exp(-Y)$ is exponentially distributed. This should get rid of one exponential in your training. The next step is to investigate $\lambda$. You seem to allow for $\lambda=0$, which will squash your exponential. Consider using a small hyperparameter $\epsilon$ in your softplus, i.e. $\ln(1+\epsilon+\exp(\cdots))$. Also it looks like $t-\mu$ can be positive or negative? This coupled with a tiny $\lambda$ will cause blowup.

Next, you've chosen weights to have mean 0 and standard deviation $0.02$. Double check that this produces sensible scales for your softplus and your $h(t)$, by predicting on a sample of your $x$'s just before running the optimization. Are your inputs $x$ normalized?

Finally, your loss function looks something like $-y_i\ln(h(x_i))-(1-y_i)\ln(h(x_i))$. You can remove numerical headaches here by manually simplifying this. If $h$ is Gumbal, then the first logorithm dissapears. If $h$ is exponential (after the above transformation), then the first logorithm will dissapear, and the second logorithm will act on a softmax (when properly rewritten). Use tensorflow's native softmax implementation for the second one, to ensure you're not blowing up.

  • $\begingroup$ For the first suggestion, do you mean I can transform y to -ln(y) so as to remove one exp? $\endgroup$
    – Munichong
    Commented Aug 12, 2018 at 1:55
  • $\begingroup$ Yes. It's easier to see in reverse. If $X$ is exponentially distributed, then $P(X\leq x)=1-e^{-\lambda x}$, so $P(e^{-X}\geq e^{-x})=1-e^{-\lambda e^{-x}}$, i.e. $e^{-X}$ is Gumbal. $\endgroup$
    – Alex R.
    Commented Aug 12, 2018 at 22:46
  • $\begingroup$ So my model uses Exp distribution (instead of Gumbel) and outputs $-ln(\hat y)$. Just wanna confirm, in this case, the loss cannot be measured by LogLoss, since $-ln(y)$ is not a probability. Maybe RMSD or anything more similar to Logloss? $\endgroup$
    – Munichong
    Commented Aug 13, 2018 at 21:23

This is probably due to one or more of the following:

  • gradients are too large; use gradient clipping.
  • learning rates are too large; use smaller learning rates.

The clue is that the loss oscillates instead of decreasing gradually early in the training. This is usually an indication that something is going wrong with the optimization routine.

  • $\begingroup$ Thanks for your reply. I tried to use gradient clipping (tf.clip_by_global_norm). The problem is still there. I printed out all values every iteration. I see that the gradient computed at the end of one iteration suddenly become all Nan. Do you know why? $\endgroup$
    – Munichong
    Commented Aug 11, 2018 at 1:45
  • $\begingroup$ The why is obvious: numerical overflow or under flow . The question is how to best limit it. If these methods don’t work, you’ll have to re-express your loss on a different scale. Alex has a good suggestion of how to do that. $\endgroup$
    – Sycorax
    Commented Aug 11, 2018 at 2:22
  • 1
    $\begingroup$ @Sycorax I wouldn't say that "the why is obvious." The OP is intelligent and is just asking why. $\endgroup$
    – Mark White
    Commented Aug 12, 2018 at 2:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.