Estimating a rate of failure/survival using only right censored data?

Question

I am trying to estimate the probability $q$ that a household with certain known covariates will move to a new home in the following year, by estimating an event rate $\lambda$ dependending on some covariates in an exponential distributon model. The data I have available comes from a large survey which includes the covariates and how long a household has lived in their current home up until the survey date (I'll call it “lvd”).

The trouble is, all reported “lvd” values are right censored survival times. Most (but likely not all) households will move some time after the survey date. If I look at a histogram of “lvd” data, I see something resembling an exponential distribution. Thinking about this distribution, if the actual event rate was very large (i.e. everybody moves almost all the time), you would expect the distribution of “lvd” to have most of its density close to 0. If the actual event rate was close to zero, you would expect a very large spread in “lvd” times. My guess was that there should be a way to estimate the event rate based purely on these right censored data, but I don’t know how. I tried to use a maximum-likelihood estimator much like in the answer to this question: [ ML estimate of exponential distribution (with censored data) ] but, I ran into the same problem as mentioned there: the MLE gives $\lambda = 0$ when all data is right censored.

Can anyone give a suggestion how one could estimate the event rate $\lambda$ (or $q$ directly) when only right censored data is available, or if this is at all possible?

(*I’m using an exponential model because I wish to apply the model to situations where I don’t know how long a household has lived in their current home, so I want some general/average event rate $\lambda$ for young vs older famillies, or married vs. single etc. Also, I’m pretty new to survival analysis and non-parametric models, so I just stuck with what I knew. If some other approach is preferred, I wouldn’t mind using that.)

Answer 1

The short version is that you can't really, at least not in a standard way. The information on the exponential distribution that you mention is in fact time to censoring, which is not such an interesting concept. From a non-parametric perspective, there is nothing you can do as models such as Cox proportional hazards put mass only at event time points (and there are none here).

If you assume a parametric model, i.e. time to moving is $Exp(\lambda)$, then the likelihood contribution of an observation at time $t$ would be $$ (\lambda)^\delta \exp(-\lambda t) $$ with $\delta = 1$ if there is an event at $t$ or 0 for censoring. With a data set of $t_1 ... t_N$ censoring events, the likelihood is $$ L = \prod_i \exp(-\lambda t_i) $$ and the log-likelihood $$ l = -\lambda \sum_i t_i $$ and you can see clearly that $\lambda = 0$ is the maximizer for this.

Which makes sense. If you had a data set on humans and the length of their lives, and you just had right censored data (only what they lived longer than), you would think that they live forever (or that they die all at any time after the follow-up - but that would not work with an exponential assumption).

Your way out of this is to go Bayesian. That is because you have a prior information on the time to moving (as you said, they move at a certain point).

For the exponential distribution, a conjugate prior is the gamma distribution for $\lambda$, so assuming $\lambda \sim gamma(\alpha, \beta)$, you would obtain your posterior as $\lambda | data \sim gamma(\alpha, \beta + \sum_it_i)$. The expectation of the posterior would be your Bayesian estimate, which is $\alpha / (\beta + \sum_it_i)$. So you can consider that.

However, keep in mind that most likely your distribution is not exponential. That is because the density of the exponential distribution is decreasing, which means that you would expect most events to take place earlier in the follow-up. There are other parametric distribution you could investigate, such as a Weibull distribution or so.