Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Apologies in advance if any of the terminology I use is incorrect. I'd welcome any correction. If what I describe as a "cut-off" goes by a different name, let me know and I can update the question.

The situation I'm interested in is this: you have independent variables $\bf{x}$ and a single dependent variable $y$. I'll leave it vague, but assume that it would be relatively straightforward getting a good regression model for these variables.

However, the model you're aiming to create is for independent variables $\bf{x}$ and dependent variable $w = min(y,a)$, where $a$ is some fixed value within the range of $y$. Equally, the data you have access to does not include $y$, only $w$.

A (somewhat unrealistic) example of this would be if you were trying to model how many years people will collect their pension for. In this case, $\bf{x}$ could be relevant information such as gender, weight, hours of exercise per week, etc. The 'underlying' variable $y$ would be life expectancy. However the variable you'd have access to and be trying to predict in your model would be $w = min(0, y-r)$ where r is the retirement age (assuming for simplicity it's fixed).

Is there a good approach for dealing with this in regression modelling?

share|improve this question
1  
I'm not certain, but this sounds like it might be approachable through some variation of survival analysis. 1) It involves censoring 2) At least in your example, it involves time. But it would be left-censored rather than right-censored (which is more common). If you agree with me, you could add the survival tag and see if anyone jumps on it. –  Peter Flom Jan 22 at 20:17
4  
@Peter It sure looks right-censored to me. Which side the censoring occurs on is of little import, because by negating the dependent variable one switches between right- and left-censoring. –  whuber Jan 22 at 20:20
    
@whuber I think you are right. But, as you say, the censoring can switch easily enough. –  Peter Flom Jan 22 at 20:31
    
The retirement example seems to call for a count data model (if you're willing to round to whole years and as long as everyone is dead by the time you run the analysis). The latent variable approach seems like stretch with this since time cannot be negative. –  Dimitriy V. Masterov Jan 22 at 23:06

1 Answer 1

up vote 9 down vote accepted

This kind of model goes by several names, depending on discipline and topic area. Common names for it are Censored Dependent Variables, Truncated Dependent Variables, Limited Dependent Variables, Survival Analysis, Tobit, and Censored Regression. I am probably leaving out several other names.

The setup you suggest where $min\{y_i,a\}$ is observed is called "right censoring," because values of $y_i$ too far to the right on the real line are censored---and instead we just see the censoring point, $a$.

One way of dealing with data like this is through the use of latent variables (and this is basically what you propose). Here is one way to proceed:

\begin{align} y_i &= x_i'\beta+\epsilon_i\\ w_i &= min\{y_i,a\}\\ \epsilon_i &\sim N(0,\sigma^2)\; iid \end{align}

Then, you can analyze this by maximum likelihood. The observations where the censoring occurs contribute $P\{y_i>a\}=\Phi(\frac{1}{\sigma}x_i'\beta-a)$ to the likelihood function, and the observations where censoring does not occur contribute $\frac{1}{\sigma}\phi((y_i-x_i'\beta)/\sigma)$ to the likelihood function. The CDF of standard normal is $\Phi$ and the density of standard normal is $\phi$. So, the likelihood function looks like:

\begin{align} L(\beta,\sigma) &= \prod_{i \in \text{censored}} \Phi(\frac{1}{\sigma}x_i'\beta-a) \prod_{i \not\in censored} \frac{1}{\sigma}\phi((y_i-x_i'\beta)/\sigma) \end{align}

You estimate the $\beta$ and $\sigma$ by maximizing this. You get standard errors as the usual maximum likelihood standard errors.

As you might imagine, this is just one approach among many.

share|improve this answer
    
+1 A worked example of the ML solution appears at stats.stackexchange.com/questions/49443. –  whuber Jan 23 at 4:48
    
@whuber That is a nice exposition. –  Bill Jan 23 at 16:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.