Apologies in advance if any of the terminology I use is incorrect. I'd welcome any correction. If what I describe as a "cut-off" goes by a different name, let me know and I can update the question.

The situation I'm interested in is this: you have independent variables $\bf{x}$ and a single dependent variable $y$. I'll leave it vague, but assume that it would be relatively straightforward getting a good regression model for these variables.

However, the model you're aiming to create is for independent variables $\bf{x}$ and dependent variable $w = \min(y,a)$, where $a$ is some fixed value within the range of $y$. Equally, the data you have access to does not include $y$, only $w$.

A (somewhat unrealistic) example of this would be if you were trying to model how many years people will collect their pension for. In this case, $\bf{x}$ could be relevant information such as gender, weight, hours of exercise per week, etc. The 'underlying' variable $y$ would be life expectancy. However the variable you'd have access to and be trying to predict in your model would be $w = \min(0, y-r)$ where r is the retirement age (assuming for simplicity it's fixed).

Is there a good approach for dealing with this in regression modelling?

  • 1
    $\begingroup$ I'm not certain, but this sounds like it might be approachable through some variation of survival analysis. 1) It involves censoring 2) At least in your example, it involves time. But it would be left-censored rather than right-censored (which is more common). If you agree with me, you could add the survival tag and see if anyone jumps on it. $\endgroup$
    – Peter Flom
    Jan 22 '14 at 20:17
  • 4
    $\begingroup$ @Peter It sure looks right-censored to me. Which side the censoring occurs on is of little import, because by negating the dependent variable one switches between right- and left-censoring. $\endgroup$
    – whuber
    Jan 22 '14 at 20:20
  • $\begingroup$ @whuber I think you are right. But, as you say, the censoring can switch easily enough. $\endgroup$
    – Peter Flom
    Jan 22 '14 at 20:31
  • $\begingroup$ The retirement example seems to call for a count data model (if you're willing to round to whole years and as long as everyone is dead by the time you run the analysis). The latent variable approach seems like stretch with this since time cannot be negative. $\endgroup$
    – dimitriy
    Jan 22 '14 at 23:06

This kind of model goes by several names, depending on discipline and topic area. Common names for it are Censored Dependent Variables, Truncated Dependent Variables, Limited Dependent Variables, Survival Analysis, Tobit, and Censored Regression. I am probably leaving out several other names.

The setup you suggest where $\min\{y_i,a\}$ is observed is called "right censoring," because values of $y_i$ too far to the right on the real line are censored---and instead we just see the censoring point, $a$.

One way of dealing with data like this is through the use of latent variables (and this is basically what you propose). Here is one way to proceed:

\begin{align} y_i &= x_i'\beta+\varepsilon_i\\ w_i &= \min\{y_i, a\}\\ \varepsilon_i &\sim N(0,\sigma^2)\; \ {\rm iid} \end{align}

Then, you can analyze this by maximum likelihood. The observations where the censoring occurs contribute $P\{y_i>a\}=\Phi(\frac{1}{\sigma}x_i'\beta-a)$ to the likelihood function, and the observations where censoring does not occur contribute $\frac{1}{\sigma}\phi((y_i-x_i'\beta)/\sigma)$ to the likelihood function. The CDF of standard normal is $\Phi$ and the density of standard normal is $\phi$. So, the likelihood function looks like:

\begin{align} L(\beta,\sigma) &= \prod_{i\ \in\ \text{censored}} \Phi\left(\frac{1}{\sigma}x_i'\beta-a\right) \prod_{i\ \not\in\ \text{censored}} \frac{1}{\sigma}\phi\big((y_i-x_i'\beta)/\sigma\big) \end{align}

You estimate the $\beta$ and $\sigma$ by maximizing this. You get standard errors as the usual maximum likelihood standard errors.

As you might imagine, this is just one approach among many.

  • 1
    $\begingroup$ +1 A worked example of the ML solution appears at stats.stackexchange.com/questions/49443. $\endgroup$
    – whuber
    Jan 23 '14 at 4:48
  • $\begingroup$ @whuber That is a nice exposition. $\endgroup$
    – Bill
    Jan 23 '14 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.