3
$\begingroup$

I would like to explore a model to predict the value of a continuous response variable, from a set (around 100) of explanatory variables. I do not want to apply PCA like feature reduction, because I want to keep my model maximally informative.

A straightforward method is to conduct a multiple linear regression on a pre-selected subset of candidate explanatory variables. However, a multiple regression requires 1) no missing measurement from any of the explanatory variables; and 2) linear relationship. In my case, I notice some explanatory variables do show strong linear relationship but with many missing values. Therefore, conducting a multiple regression including all candidate explanatory variables is not feasible.

My questions are:

1) Are there any machine learning techniques that can handle continuous response variable, but with many NA in many explanatory variables? The relationships can be non-linear.

2) Is regression tree a proper method to use in my case?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Random forests and gradient boosted machines (both based on regression trees, but improvements on them) can do this, but the treatment of NAs (typically based on imputation) can be tricky and is worth thinking about. $\endgroup$
    – mkt
    Commented Dec 6, 2017 at 15:31
  • $\begingroup$ Thanks @mkt for the reply. Do you know how regression trees or random forest treat NAs? Seems that they treat NA as zero? $\endgroup$
    – tiantianchen
    Commented Dec 7, 2017 at 9:06

1 Answer 1

Reset to default
2
$\begingroup$

Random forests (RFs) and gradient boosting machines (GBMs, also called boosted regression trees) can do this. Both are methods based on ensembles of regression trees. I should note that there are certainly other methods that can be used for problems such as this, but I'll focus on these since I am somewhat familiar with them.

In the case of RFs, there are multiple approaches used that depend on the implementation: NAs may either be excluded or handled by imputation (based on the mean/median value, or value of similar points).

GBMs (perhaps not universally, but in the implementations I've seen) handle NAs in a different way: through surrogate splits. This is explained in Tierney et al. (2015) as:

[Surrogate split] means that when a value for a variable is missing, and that variable needs to be used to determine a split, an alternative variable that is highly correlated with the missing variable is used to determine the direction of the split.

My impression is that the GBMs do a better job of handling missing data, but I don't have a lot of evidence for that claim.

Tierney, N. J., Harden, F. A., Harden, M. J., & Mengersen, K. L. (2015). Using decision trees to understand structure in missing data. BMJ Open, 5(6), e007450. http://doi.org/10.1136/bmjopen-2014-007450

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.