39
$\begingroup$

I am learning survival analysis from this post on UCLA IDRE and got tripped up at section 1.2.1. The tutorial says:

... if the survival times were known to be exponentially distributed, then the probability of observing a survival time ...

Why are survival times assumed to be exponentially distributed? It seems very unnatural to me.

Why not normally distributed? Say suppose we are investigating some creature's life span under certain condition (say number of days), should it be more centered around some number with some variance (say 100 days with variance 3 days)?

If we want time to be strictly positive, why not make normal distribution with higher mean and very small variance (will have almost no chance to get negative number.)?

Improve this question
$\endgroup$
  • 10
    $\begingroup$ Heuristically, I cannot think of the normal distribution as an intuitive way to model failure time. It's never cropped up in any of my applied work. They are always skewed very far right. I think normal distributions heuristically come about as a matter of averages, whereas survival times heuristically come about as a matter of extrema such as the effect of a constant hazard being applied to a sequence of parallel or series components. $\endgroup$ – AdamO Mar 17 '17 at 15:43
  • 6
    $\begingroup$ I agree with @AdamO about the extreme distributions inherent to survival and time to failure. As others have noted, exponential assumptions have the advantage of being tractable. The biggest problem with them is the implicit assumption of a constant rate of decay. Other functional forms are possible and come as standard options depending on the software, e.g., generalized gamma. Goodness of fit tests can be employed to test differing functional forms and assumptions. The best text on survival modeling is Paul Allison's Survival Analysis Using SAS, 2nd ed. Forget SAS-it's an excellent review $\endgroup$ – Mike Hunter Mar 17 '17 at 18:23
  • 8
    $\begingroup$ I would note that the very first word in your quote is "if" $\endgroup$ – Fomite Mar 17 '17 at 19:34

10 Answers 10

Active Oldest Votes
42
$\begingroup$

Exponential distributions are often used to model survival times because they are the simplest distributions that can be used to characterize survival / reliability data. This is because they are memoryless, and thus the hazard function is constant w/r/t time, which makes analysis very simple. This kind of assumption may be valid, for example, for some kinds of electronic components like high-quality integrated circuits. I'm sure you can think of more examples where the effect of time on hazard can safely be assumed to be negligible.

However, you are correct to observe that this would not be an appropriate assumption to make in many cases. Normal distributions can be alright in some situations, though obviously negative survival times are meaningless. For this reason, lognormal distributions are often considered. Other common choices include Weibull, Smallest Extreme Value, Largest Extreme Value, Log-logistic, etc. A sensible choice for model would be informed by subject-area experience and probability plotting. You can also, of course, consider non-parametric modeling.

A good reference for classical parametric modeling in survival analysis is: William Q. Meeker and Luis A. Escobar (1998). Statistical Methods for Reliability Data, Wiley

Improve this answer
$\endgroup$
  • $\begingroup$ could you elaborate more on " hazard function is constant w/r/t time"? $\endgroup$ – Haitao Du Mar 18 '17 at 6:14
  • 4
    $\begingroup$ @hxd1011: Presumably by "hazard function" the author is referring to the function $r_X$ given by $r_X(t) = f_X(t) / \bar F_X(t)$, where $f_X$ is the pdf of $X$ and $\bar F_X$ is the tail of $X$ ($\bar F_X(t) = 1 - F_X(t) = \int_t^\infty f_X(x) \, dx$). This is also called the failure rate. The observation is that for $\operatorname{Exp}(\lambda)$, the failure rate is $r(t) =(\lambda e^{-\lambda t}) / (e^{-\lambda t}) = \lambda$, which is constant. Furthermore, it is not hard to show that only the exponential distribution has this property. $\endgroup$ – wchargin Mar 19 '17 at 16:42
23
$\begingroup$

To add a bit of mathematical intuition behind how exponents pop up in survival distributions:

The probability density of a survival variable is $f(t) = h(t)S(t)$, where $h(t)$ is the current hazard (risk for a person to "die" this day) and $S(t)$ is the probability that a person survived until $t$. $S(t)$ can be expanded as the probability that a person survived day 1, and survived day 2, ... up to day $t$. Then: $$ P(survived\ day\ t)=1-h(t)$$ $$ P(survived\ days\ 1, 2, ..., t) = (1-h(t))^t$$ With constant and small hazard $\lambda$, we can use: $$ e^{-\lambda} \approx 1-\lambda$$ to approximate $S(t)$ as simply $$ (1-\lambda)^t \approx e^{-\lambda t} $$ , and the probability density is then $$ f(t) = h(t)S(t) = \lambda e^{-\lambda t}$$

Disclaimer: this is in no way an attempt at a proper derivation of the pdf - I just figured this is a neat coincidence, and welcome any comments on why this is correct/incorrect.

EDIT: changed the approximation per advice by @SamT, see comments for discussion.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ +1 this helped me to understand more on properties of exponential distribution. $\endgroup$ – Haitao Du Mar 17 '17 at 16:51
  • 1
    $\begingroup$ Could you explain your penultimate line? It says $S(t) = ...$, so the left hand side is function of $t$; moreover, so is the right. However, the two middle terms are functions of $\lambda$ (as is the right hand side), but not functions of $t$. Moreover, the approximation $(1+x/n)^n ~ e^{x}$ only holds for $x = o(\sqrt{n})$. It's certainly not true that $\lim_{t \to \infty} (1-\lambda t/t)^t = e^{-\lambda t}$ -- it's not even approximately true for large $t$. I guess this is just a notational mistake you've made though...? $\endgroup$ – Sam OT Mar 17 '17 at 17:50
  • $\begingroup$ @SamT - thanks for the comment, edited. Coming from an applied background, I very much welcome any corrections, esp. on notation. Passing to the limit wrt $t$ was certainly not needed there, but I still believe the approximation holds for small $\lambda$, as are typically encountered in survival models. Or would you say there's something else that coincidentally makes this approximation hold? $\endgroup$ – juod Mar 17 '17 at 20:08
  • 1
    $\begingroup$ Looks better now :) -- the issue is that while $\lambda$ may be small it's not true that $\lambda t$ is necessarily small; as such, you can't use the approximation $$(1+x/n)^n \approx e^x$$ (directly): it's not even "you can in applied maths but can't in pure"; it just doesn't hold at all. However, we can get around this: we do have that $\lambda$ is small, so we can get there directly, writing $$e^{-\lambda t} = \big(e^{-\lambda}\big)^t \approx \big(1-\lambda)^t.$$ Of course, $\lambda = \lambda t / t$, so we can then deduce that $$e^{-\lambda t} \approx \big(1 - \lambda t / t\big)^t.$$ $\endgroup$ – Sam OT Mar 17 '17 at 20:14
  • $\begingroup$ Being applied, you may feel this is being slightly picky, but the point is that the reasoning wasn't valid; similar invalid steps may not happen to be true. Of course, as someone applied, you may be happy to make this step, find it holds in the majority of cases and not worry about the specifics! As someone who does pure maths, this is out of the question for me, but I understand that we need both pure and applied! (And particularly in stats it's good not to get bogged down in pure technicalities.) $\endgroup$ – Sam OT Mar 17 '17 at 20:16
11
$\begingroup$

You'll almost certainly want to look at reliability engineering and predictions for thorough analyses of survival times. Within that, there are a few distributions which get used often:

The Weibull (or "bathtub") distribution is the most complex. It accounts for three types of failure modes, which dominate at different ages: infant mortality (where defective parts break early on), induced failures (where parts break randomly throughout the life of the system), and wear out (where parts break down from use). As used, it has a PDF which looks like "\__/". For some electronics especially, you might hear about "burn in" times, which means those parts have already been operated through the "\" part of the curve, and early failures have been screened out (ideally). Unfortunately, Weibull analysis breaks down fast if your parts aren't homogeneous (including use environment!) or if you are using them at different time scales (e.g. if some parts go directly into use, and other parts go into storage first, the "random failure" rate is going to be significantly different, due to blending two measurements of time (operating hours vs. use hours).

Normal distributions are almost always wrong. Every normal distribution has negative values, no reliability distribution does. They can sometimes be a useful approximation, but the times when that's true, you're almost always looking at a log-normal anyway, so you may as well just use the right distribution. Log-normal distributions are correctly used when you have some sort of wear-out and negligible random failures, and in no other circumstances! Like the Normal distribution, they're flexible enough that you can force them to fit most data; you need to resist that urge and check that the circumstances make sense.

Finally, the exponential distribution is the real workhorse. You often don't know how old parts are (for example, when parts aren't serialized and have different times when they entered into service), so any memory-based distribution is out. Additionally, many parts have a wearout time that is so arbitrarily long that it's either completely dominated by induced failures or outside the useful time-frame of the analysis. So while it may not be as perfect a model as other distributions, it just doesn't care about things which trip them up. If you have an MTTF (population time/failure count), you have an exponential distribution. On top of that, you don't need any physical understanding of your system. You can do exponential estimates just based on observed part MTTFs (assuming a large enough sample), and they come out pretty dang close. It's also resilient to causes: if every other month, someone gets bored and plays croquet with some part until it breaks, exponential accounts for that (it rolls into the MTTF). Exponential is also simple enough that you can do back-of-the-envelope calculations for availability of redundant systems and such, which significantly increases its usefulness.

Improve this answer
$\endgroup$
  • 3
    $\begingroup$ This is a good answer, but note that the Weibull distribution is not "the most complex" parametric distribution for survival models. I'm not sure if there could be such a thing, but certainly relative to the Weibull there is the generalized Gamma distribution, & the generalized F distribution, both of which can take the Weibull as a special case by setting parameters to 0. $\endgroup$ – gung - Reinstate Monica Mar 17 '17 at 19:49
  • $\begingroup$ It's the most complex one commonly used in reliability engineering (first paragraph :) I don't disagree with your point, but I also have never seen either actually used (write-ups of how they could be used, yes. Actual implementation, no) $\endgroup$ – fectin Mar 18 '17 at 3:55
9
$\begingroup$

Some ecology might help answer the "Why" behind this question.

The reason why exponential distribution is used for modeling survival is due to the life strategies involved in organisms living in nature. There's essentially two extremes with regard to survival strategy with some room for the middle ground.

Here's an image that illustrates what I mean (courtesy of Khan Academy):

https://www.khanacademy.org/science/biology/ecology/population-ecology/a/life-tables-survivorship-age-sex-structure

This graph plots surviving individuals on the Y axis, and "percentage of maximum life expectancy" (a.k.a. approximation of the individual's age) on the X axis.

Type I is humans, which model organisms which have an extreme level of care of their offspring ensuring very low infant mortality. Often these species have very few offspring because each one takes a large amount of the parents time and effort. The majority of what kills Type I organisms is the type of complications that arise in old age. The strategy here is high investment for high payoff in long, productive lives, if at the cost of sheer numbers.

Conversely, Type III is modeled by trees (but could also be plankton, corals, spawning fish, many types of insects, etc) where the parent invests relatively little in each offspring, but produces a ton of them in the hopes that a few will survive. The strategy here is "spray and pray" hoping that while most offspring will be destroyed relatively quickly by predators taking advantage of easy pickings, the few that survive long enough to grow will become increasingly difficult to kill, eventually becoming (practically) impossible to be eaten. All the while these individuals produce huge numbers of offspring hoping that a few will likewise survive to their own age.

Type II is a middling strategy with moderate parental investment for moderate survivability at all ages.

I had an ecology professor who put it this way:

"Type III (trees) is the 'Curve of Hope', because the longer an individual survives, the more likely it becomes that it will continue to survive. Meanwhile Type I (humans) is the 'Curve of Despair', because the longer you live, the more likely it becomes that you will die."

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ This is interesting, but note that for humans, before modern medicine (& still in some places in the world today), infant mortality is very high. Baseline human survival is often modeled with "bathtub hazard". $\endgroup$ – gung - Reinstate Monica Mar 17 '17 at 19:16
  • 1
    $\begingroup$ @gung Absolutely, this is a broad generalization and there are variations within humans of different regions and time periods. The main difference is clearer when you're comparing extremes, i.e. Western human families (~2.5 children per pair, most of which don't die in infancy) vs corals or spawning fish (millions of eggs released per mating cycle, most of which die due to being eaten, starvation, hazardous water chemistry, or simply failing to drift into a habitable destination) $\endgroup$ – CaffeineConnoisseur Mar 17 '17 at 19:18
  • 2
    $\begingroup$ While I'm all for explanations from ecology, I'll note assumptions like this are also made for things like hard drives and aircraft engines. $\endgroup$ – Fomite Mar 17 '17 at 19:35
9
$\begingroup$

To answer your explicit question, you cannot use the normal distribution for survival because the normal distribution goes to negative infinity, and survival is strictly non-negative. Moreover, I don't think it's true that "survival times are assumed to be exponentially distributed" by anyone in reality.

When survival times are modeled parametrically (i.e., when any named distribution is invoked), the Weibull distribution is the typical starting place. Note that the Weibull has two parameters, shape and scale, and that when shape = 1, the Weibull simplifies to the exponential distribution. A way of thinking about this is that the exponential distribution the simplest possible parametric distribution for survival times, which is why it is often discussed first when survival analysis is being taught. (By analogy, consider that we often begin teaching hypothesis testing by going over the one-sample $z$-test, where we pretend to know the population SD a-priori, and then work up to the $t$-test.)

The exponential distribution assumes that the hazard is always exactly the same, no matter how long a unit has survived (consider the figure in @CaffeineConnoisseur's answer). In contrast, when the shape is $>1$ in the Weibull distribution, it implies that hazards increase the longer you survive (like the 'human curve'); and when it is $<1$, it implies hazards decrease (the 'tree').

Most commonly, survival distributions are complex and not well fit by any named distribution. People typically don't even bother trying to figure out what distribution it might be. That's what makes the Cox proportional hazards model so popular: it is semi-parametric in that the baseline hazard can be left completely unspecified but the rest of the model can be parametric in terms of its relationship to the unspecified baseline.

Improve this answer
$\endgroup$
  • 4
    $\begingroup$ "Moreover, I don't think it's true that "survival times are assumed to be exponentially distributed" by anyone in reality." I've actually found it to be quite common in epidemiology, usually implicitly. $\endgroup$ – Fomite Mar 17 '17 at 23:16
  • 1
    $\begingroup$ @gung, could you kindly explain - it is semi-parametric in that the baseline hazard can be left completely unspecified but the rest of the model can be parametric in terms of its relationship to the unspecified baseline $\endgroup$ – Gaurav Singhal Jun 28 '18 at 10:16
6
$\begingroup$

This doesn't directly answer the question, but I think it's very important to note, and does not fit nicely into a single comment.

While the exponential distribution has a very nice theoretical derivation, and thus assuming the data produced follows the mechanisms assumed in the exponential distribution, it should theoretically give optimal estimates, in practice I've yet to run into a dataset where the exponential distribution produces even close to acceptable results (of course, this is dependent on the data types I've analyzed, almost all biological data). For example, I just looked at fitting a model with a variety of distributions using the first data set I could find in my R-package. For model checking of the baseline distribution, we typically compare against the semi-parametric model. Take a look at the results.

Survival Curves

Of the Weibull, log-logistic and log-normal distribution, there's not an absolute clear victor in terms of appropriate fit. But there's a clear loser: the exponential distribution! It's been my experience that this magnitude of mis-fitting is not exceptional, but rather the norm for the exponential distribution.

Why? Because the exponential distribution is a single parameter family. Thus, if I specify the mean of this distribution, I've specified all other moments of the distribution. These other families are all two parameter families. Thus, there's a lot more flexibility in those families to adapt to the data itself.

Now keep in mind that the Weibull distribution has the exponential distribution as a special case (i.e. when the shape parameter = 1). So even if the data truly is exponential, we only add a little more noise to our estimates by using a Weibull distribution over an exponential distribution. As such, I would just about never recommend using the exponential distribution to model real data (and I'm curious to hear if any readers have an example of when it's actually a good idea).

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ I am not convinced of this answer: 1) "using the first data set I could find in my R-package"... Really? ... on stats.stackexchange? One random sample and we draw general conclusions? 1b) For models where the failure time tends to be distributed around a given value (like people's life), clearly the distributions like Gamma, Weibull, etc are more suited; when events are equally probable an exponential distribution is more suited. I bet your "first data set" above is of the first kind. 2) All other models have 2 parameters, one should use e.g. the Bayes factor to compare the models. $\endgroup$ – Luca Citi Mar 19 '17 at 22:57
  • 2
    $\begingroup$ @LucaCiti: "the first data set in my R-package" means the first dataset in the R-package that I published (icenReg). And I did note that my experience with the exponential distribution always having a poor fit was dependent on the type of data I've analyzed; almost exclusively biological data. Finally, as I stated in the end, I'm very curious to hear real applied examples where there's a convincing reason to use the exponential distribution, so if you have one, please share. $\endgroup$ – Cliff AB Mar 20 '17 at 1:47
  • 1
    $\begingroup$ A scenario when you might want to use the exponential distribution would be when (a) you had a lot of historic data that showed that the data really was well approximated with an exponential distribution and (b) you needed to make inference with small samples (i.e. n < 10). But I don't know of any real applications like this. Maybe in some sort of manufacturing quality control problem? $\endgroup$ – Cliff AB Mar 20 '17 at 1:53
  • 1
    $\begingroup$ Hi Cliff, thanks for taking the time to reply to my comment. I think roughly speaking a distribution like the Weibull fits better situations corresponding to questions like "what is the life time of individual x in my sample" or "when is neuron x going to fire again" or "when is firefly x going to flash again". Conversely, an exponential distribution models questions like "when is the next death expected to happen in my population", "when is the next neuron going to fire" or "when is a firefly in the swarm going to flash" $\endgroup$ – Luca Citi Mar 20 '17 at 8:54
  • $\begingroup$ @LucaCiti; ha, just got that your earlier poke was a joke about making an inference with n = 1. Don't know how I missed it the first time. In my defense, if we have theory that says the estimator should be asymptotically normal yet it's 4+ standard deviations away from the other asymptotically normal estimates, then we can! But in all seriousness, it's not that one plot that convinced me, but seeing that same level of deviation consistently. I may get blocked if I spam 20+ plots of bad exponential fits though. $\endgroup$ – Cliff AB Mar 21 '17 at 14:04
4
$\begingroup$

Another reason why the exponential distribution crops up often to model interval between events is the following.

It is well known that, under some assumptions, the sum of a large number of independent random variables will be close to a Gaussian distribution. A similar theorem holds for renewal processes, i.e. stochastic models for events that occur randomly in time with I.I.D. inter-event intervals. In fact, the Palm–Khintchine theorem states that the superposition of a large number of (not necessarily Poissonian) renewal processes behaves asymptotically like a Poisson process. The inter-event intervals of a Poisson process are exponentially distributed.

Improve this answer
$\endgroup$
3
$\begingroup$

tl;dr- An expontential distribution is equivalent to assuming that individuals are as likely to die at any given moment as any other.

Derivation

  1. Assume that a living individual is as likely to die at any given moment as at any other.

  2. So, the death rate $-\frac{\text{d}P}{\text{d}t}$ is proportional to the population, $P$.

$$-\frac{\text{d}P}{\text{d}t}{\space}{\propto}{\space}P$$

  1. Solving on WolframAlpha shows:

$$P\left(t\right)={c_1}{e^{-t}}$$

So, the population follows an exponential distribution.

Math note

The above math is a reduction of a first-order ordinary differential equation (ODE). Normally, we would also solve for $c_0$ by noting the boundary condition that population starts at some given value, $P\left(t_0\right)$, at start-time $t_0$.

Then the equation becomes: $$P\left(t\right)={e^{-t}}P\left({t_0}\right).$$

Reality check

The exponential distribution assumes that people in the population tend to die at the same rate over time. In reality, death rates will tend to vary for finite populations.

Coming up with better distributions involves stochastic differential equations. Then, we can't say that there's a constant death likelihood; rather, we have to come up with a distribution for each individual's odds of dying at any given moment, then combine those various possibility trees together for the entire population, then solve that differential equation over time.

I can't recall having seen this done in anything online before, so you probably won't run into it; but, that's the next modeling step if you want to improve upon the exponential distribution.

Improve this answer
$\endgroup$
3
$\begingroup$

(Note that in the part you quoted, the statement was conditional; the sentence itself didn't assume exponential survival, it explained a consequence of doing so. Nevertheless assumption of exponential survival are common, so it's worth dealing with the question of "why exponential" and "why not normal" -- since the first is pretty well covered already I'll focus more on the second thing)

Normally distributed survival times don't make sense because they have a non-zero probability of the survival time being negative.

If you then restrict your consideration to normal distributions that have almost no chance of being near zero, you can't model survival data that has a reasonable probability of a short survival time:

survival time distributions -- normal mean 100 sd 10 vs a particular distribution with mean 100 and sd 42 which has more than 20% probability of survival times between 0 and 50

Maybe once in a while survival times which have almost no chance of short survival times would be reasonable, but you need distributions that make sense in practice -- usually you observe short and long survival times (and anything in between), with typically a skewed distribution of survival times). An unmodified normal distribution will rarely be useful in practice.

[A truncated normal might more often be a reasonable rough approximation than a normal, but other distributions will often do better.]

The constant-hazard of the exponential is sometimes a reasonable approximation for survival times.. For example, if "random events" like accident are a major contributor to death-rate, exponential survival will work fairly well. (Among animal populations for example, sometimes both predation and disease can act at least roughly like a chance process, leaving something like an exponential as a reasonable first approximation to survival times.)

One additional question related truncated normal: if normal is not appropriate why not normal squared (chi sq with df 1)?

Indeed that might be a little better ... but note that that would correspond to an infinite hazard at 0, so it would only occasionally be useful. While it can model cases with a very high proportion of very short times, it has the converse problem of only being able to model cases with typically much shorter than average survival (25% of survival times are below 10.15% of the mean survival time and half of the survival times are less than 45.5% of the mean; that is median survival is less than half the mean.)

Let's look at a scaled $χ^2_1$ (i.e. a gamma with shape parameter $\frac12$):

Similar plot to before, but also with density of a variate that is 100 times a chi-squared(1); it's got a high peak at 0 and a very heavy tail -- the mean is 100 but the sd is about 141 and the median is about 45.

[Maybe if you sum two of those $χ^2_1$ variates... or maybe if you considered noncentral $χ^2$ you would get some suitable possibilities. Outside of the exponential, common choices of parametric distributions for survival times include Weibull, lognormal, gamma, log-logistic among many others ... note that the Weibull and the gamma include the exponential as a special case]

Improve this answer
$\endgroup$
  • $\begingroup$ thanks, i have been waiting to your answer since yesterday :). One additional question related truncated normal: if normal is not appropriate why not normal squared (chi sq with df 1)? $\endgroup$ – Haitao Du Mar 19 '17 at 1:31
  • $\begingroup$ Indeed that might be a little better ... but note that that would correspond to an infinite hazard at 0 -- so it would only occasionally be useful. It has the converse problem of only modelling cases with typically much shorter than average survival (25% of survival times are below 10.15% of the mean survival time and half of the survival times are less than 45.5% of the mean) Maybe if you sum two of those $\chi^2_1$ variates you could get a less surprising hazard function. . .;P $\endgroup$ – Glen_b Mar 19 '17 at 2:01
  • $\begingroup$ again thank you for education my the intuition behind things. I have seen too much recipe level tutorials and people doing things without knowing why. CV is a great place to learn. $\endgroup$ – Haitao Du Mar 19 '17 at 2:08
1
$\begingroup$

If we want time to be strictly positive, why not make normal distribution with higher mean and very small variance (will have almost no chance to get negative number.)?

Because

  1. that still has a nonzero probability of being negative, so it's not strictly positive;

  2. the mean and variance are something that you can measure from the population you're trying to model. If your population has mean 2 and variance 1, and you model it with a normal distribution, that normal distribution will have substantial mass below zero; if you model it with a normal distribution with mean 5 and variance 0.1, your model obviously has very differnt properties to the thing it's supposed to model.

The normal distribution has a particular shape, and that shape is symmetrical about the mean. The only way to adjust the shape are to move it right and left (increase or decrease the mean) or to make it more or less spread out (increase or decrease the variance). This means that the only way to get a normal distribution where most of the mass is between two and ten and only a tiny amount of the mass is below zero, you need to put your mean at, say, six (the middle of the range) and set the variance small enough that only a tiny fraction of samples are negative. But then you'll probably find that most of your samples are 5, 6 or 7, whereas you were supposed to have quite a lot of 2s, 3s, 4s, 8s, 9s and 10s.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.