I am working on a binary classification task with a pretty straightforward input set of numeric features. One of these features is particularly good, but it cannot be used in real life because it's a measure that is obtained after the fact has occurred. Is it possible to predict this measure based on the other features, and then build a model including this predicted measure?

In more detail, I am building a classifier for this challenge from the UCI repo: https://archive.ics.uci.edu/ml/datasets/bank+marketing

The feature that cannot be used is the call duration because one can't know how long a call will last before it takes place. So I am wondering, could I build a regression model or at least a binned classifier to predict how long a call will last before it takes place, then feed this prediction to the model and replace the provided call duration feature?

  • 2
    $\begingroup$ That's basically a form of stacking, so yes. But you have to be very careful about how you do your train-test splits to make certain you haven't included info from that variable somewhere you shouldn't have. $\endgroup$ – Dan Jan 4 '19 at 18:04
  • $\begingroup$ Thank you for your reply! In other words, I will have two models; model 1 doesn't include the extra feature, and model 2 will. If I am understanding correctly, you are saying model 2 should use the same train-test split from model 1? $\endgroup$ – Odisseo Jan 4 '19 at 18:08
  • $\begingroup$ I suggest this, let's say you have your target feature $T$ and your other feature you can't use normally, $T'$. You can make two models, $T = F(x)$ and $T' = F'(x)$ using the same training/validation data. But then you'll want to stack these into a third model, so $T'' = F''(T,T')$. Now for this model $F''$ you have to use a different (future) train/valid set. $\endgroup$ – Dan Jan 4 '19 at 18:16
  • $\begingroup$ If instead you want to build a model $T'' = F''(T',x)$, then you need to use one dataset for building $T'=F(x)$ and future separate date for $T'' = F''(T',x)$. $\endgroup$ – Dan Jan 4 '19 at 18:22
  • $\begingroup$ But what if all the features (except call duration) are not good in predicting call duration ? Will that predicted call duration feature will be of any help in predicting actual Target variable? $\endgroup$ – Harshit Mehta Jan 4 '19 at 19:45

Let's formulate a problem. You have two sets of features $X_t$ and $Z_t$, where the former is available in future $X_{t+1}$, while the latter is not $Z_{t+1}=?$

You want to forecast some quantity $Y_t$ conditional on these features: $\hat Y_{t+1}=f(X_{t+1},Z_{t+1})$. The trouble is that $Z_{t+1}$ is not known, so you suggest to first obtain $\hat Z_{t+1}=g(X_{t+1},Z_t)$, then plug it to your first model $\hat Y_{t+1}=f(X_{t+1},g(X_{t+1},Z_t))$ Now you can use information that is available in future, i.e. $X_{t+1}$.

First, this can be done, and is done in practice.

However, conceptually, it is similar to simply building a model on what's available in future: $$Y_{t+1}=h(X_{t+1})$$

So, isn't it better to simply do the second model $h()$, instead of the two step approach with $f()$? It depends. On one hand the second approach is simpler, and thus can be more robust. On the other hand, the first approach may allow you to capture something that is not easy to incorporate in the second model.

I run into this issue all the time, and pick different path case by case.

Here's a trivial example where you want to do the second approach. Suppose, you're limited to linear modeling: $y=X\beta_x+Z\beta_z$ and $Z=X\beta_{zx}$, then you have $y=X\beta_x+X\beta_{zx}\beta_z=X(\beta_x+\beta_{zx}\beta_z)$ This is equivalent to the second approach of modeling on just $X_t$, so you don't bother and do $y=X\beta$ directly.

  • $\begingroup$ Thank you for the formalization, this makes my question clearer even to me. Unfortunately, I am going to have to go with Option 1 because that variable is very good. Would you recommend a particular approach? What seems to have worked in your experience? Would you recommend stacking? Thanks again! $\endgroup$ – Odisseo Jan 4 '19 at 18:18
  • $\begingroup$ @Odisseo, see my update to answer. Basically, in linear setup you have to be careful not to end up with a situation where it's waste of time to model in two steps $\endgroup$ – Aksakal Jan 4 '19 at 18:21
  • $\begingroup$ Thanks, this makes sense. Aside from using g(X,Z), I think it would also be helpful to throw in more models that use the known variable X, each using a different time window X's into f(X, g). So the final ensemble would be f(X_30days, X_60days, X..., Z). Do you think this would be a good idea? Basically, doing your option 1 in conjunction with an approach similar to what is described here: stats.stackexchange.com/questions/47950/… $\endgroup$ – Odisseo Jan 4 '19 at 18:38
  • $\begingroup$ @Odisseo, in my answer $X_t$ really means $X_{s\le t}$, i.e. all information known at time $t$ and earlier. So, you can have any number of lags that make a sense. Also, you can have dynamic models where $\hat Y_{t+1}=f(X_t,Y_t)$ etc. I kept the notation simple to get to the root of the problem. $\endgroup$ – Aksakal Jan 4 '19 at 18:50
  • $\begingroup$ To test this idea and create a model for Z(X), I created a regression model but unfortunately I realized during implementation that have very poor predictors and I never got above a 0.3 R2. Is there anything else you may recommend doing in this situation? Seems like I am running out of options. Thanks! $\endgroup$ – Odisseo Jan 9 '19 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.