Classification followed by regression?

Question

I have the following problem:

I have a dataset for which my observations have a bunch of features and a continuous response (regression problem). However, some of my observations (about a fourth of them) do not have a response. The features should also be predictive of those who have a response and those who do not (classification problem). Furthermore, a small response value can be interpreted similarly to those who do not have a response.

How would you go with that problem? Would you simply do a regression on the observations that have a response. Although this would throw away the information from the observations that do not have a response.

Or would you first do classification: predict if an observation has a response or not. And if yes, do regression on those. But then, how do you measure the cost if you make a wrong classification.

Answer 1

A model that uses features both to predict the probability of a response and to predict the value of a response if it occurs is a hurdle model. Hurdle models are typically encountered with count data, but the same approach can be used to model a continuous linear response once the hurdle is crossed.

This page has an extensive discussion in comments with a superb detailed answer related to this issue, in the context of count data. The takeaway is that it is OK to model separately the hurdle-crossing process and the subsequent regression, as "the log-likelihood [of the combined model] can be decomposed into two parts that can be maximized separately" for estimating the coefficients.

Under the standard assumptions about the distribution of error terms in linear regression, maximum likelihood estimation produces the same result as ordinary least squares. Thus separate modeling of hurdle and subsequent regression is OK.

Do, however, think about whether standard linear regression is appropriate for the responses in your situation. For example, if all of your responses are necessarily positive you might want to consider some type of generalized linear model. This page illustrates this approach with gamma regression.