3
$\begingroup$

In regards to feature engineering for machine learning models. I would like to engineer a feature that encodes the following:

  • value can be true and if so it will measure a numeric (maybe continuous) e.g. 'weeks since transaction', 'time since marriage'
  • however, it is possible that this event never occurred or this value doesn't apply. e.g. transaction never occurred, person never married

How might I design this feature. I can see a couple of options:

  1. Have a continuous positive variable, but a negative value for the 'false' values
  2. Have a continuous variable with NA for the 'false' values
  3. Use two separate features in some way.

My guess is 1. runs afoul of the mechanics of most algorithms, 2. doesn't work well with logistic regression without imputing something for the NA, and I don't know about 3. (Is there a name for this type of variable?)

Improve this question
$\endgroup$
  • 1
    $\begingroup$ Welcome to the world of zip models (zero inflated probabilities). It's a hierarchical process: the top level is a discrete variable (married/single) and the lower level is continuous in the first case and a point mass in the second. You need to model it in two pieces - select out the marrieds, then model the length of marriage. You may have to "roll our own" regarding the machine learning algorithm. I doubt canned software will do it. $\endgroup$ – Placidia Sep 1 '15 at 19:19
  • $\begingroup$ @Placidia: I think "feature" means "predictor" rather than "response". $\endgroup$ – Scortchi - Reinstate Monica♦ Sep 2 '15 at 11:36
  • 1
    $\begingroup$ @Scortchi Good point. I have answered the question for independent variables. $\endgroup$ – Placidia Sep 2 '15 at 12:30
  • $\begingroup$ (1) Rather arbitrary in general, regardless of algorithms' mechanics. How'd you choose which negative value to use? (2) Imputation would be useful if you're interested in the "latent" model with no missing data, either because it's interesting or you want use it to predict with future complete data. Don't think this applies to these examples. $\endgroup$ – Scortchi - Reinstate Monica♦ Sep 2 '15 at 13:20
3
$\begingroup$

This sort of thing happens a lot in survey data. Question 1: Are you married. Question 2: If you answered yes to question 1, how long have you been married? I feel your pain.

I usually go with option 3: I code a binary for yes/no bit and then a continuous variable for the "how long have you been married" bit. Note that the continuous part of option 3 is the same as option 2. Option 1 involves "making the data up", which I try not to do.

First, I analyse the full data set with just the binary variable and hope that this gives me something useful. At a minimum, it's a good filter. If marital status does not affect the outcome, then length of marriage may not affect it either. Problem solved.

If I'm interested in "length of marriage", I'm going to select out those cases that are married and limit my analysis to them. If I believe that "length of marriage" affects my response, and I'm interested in studying that issue, unmarried people really have no information to give me on that point. So I exclude them for that piece.

There is an option 4 that sometimes works, and that is to build an ordinal variable that combines both portions. Recode 1 for unmarried, 2 for married for less than five years; 3 for married for five years or more. Obviously, this only works when the "no" answer can be viewed as a lower bound to the "yes" case. This won't work, for example, if the questions were Q1, did you take Prof. Smith's machine learning course? Q2. If "Yes", rate it on a scale of 1 through 10.

There is a reason why machine learning algorithms collapse under the kind of data you describe. Basically, you are dealing with two different sample spaces -- one space for the unmarried people; one space for the married people, about whom we have additional information. So the structure of the problem is different for the two population groups. Statistical models assume one sample space, one sigma algebra, one probability measure per customer. You actually have two models going here, and I am not aware of any clean way to bridge that gap.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ (+1) But I think you undersell option 3 rather. The binary indicator & the continuous variable (by the way, you didn't say what to do with this when the binary indicator is "no" - setting it to any constant value is fine) neatly encode all the original information. In regression you can straight away include each as a separate term, or create a polynomial or spline basis for the continuous part, as well as using one or both in interaction terms (not the continuous variable alone, of course). Machine learning feature encoding gives some detail. $\endgroup$ – Scortchi - Reinstate Monica♦ Sep 2 '15 at 12:51
  • 1
    $\begingroup$ In the OP's situation, the data are not missing, they are not applicable, so not imputable. In a Bayesian situation, you might treat the "no's" as uninformative (so little change to the posterior) and have the continuous outcome (where applicable) provoke greater change. $\endgroup$ – Placidia Sep 2 '15 at 13:45
  • $\begingroup$ You're quite right (the linked question was about missing data so it's really only the 1st paragraph of my answer that applies here). $\endgroup$ – Scortchi - Reinstate Monica♦ Sep 2 '15 at 13:49
  • $\begingroup$ I think option 3 really doesn't work for me because I'm using logistic regression which wont work with NA's (afaik), and as said above, I don't know if I can really impute anything for these values without collapsing into one of the other options. Also slicing doesn't really work for me. Unmarried, and length of marriage both contain information I would like in the model.I do like your option 4. It's not ideal because it doesn't really distinguish between those who have been married 0 to 4 years, say, and those who never been married. But I think it's the best solution, thanks. $\endgroup$ – Joe Sep 2 '15 at 19:28
  • $\begingroup$ @Joe: Do read the answer carefully. Option 3 involves coding the NA's with a dummy variable, which works fine with logistic regression. Option 4 does distinguish between the unmarried, coded with '1', & the recently married, coded with '2'. (It's perhaps not clear how it's proposed here to treat the new ordinal predictor - using dummies for each category is just Option 3 + binning of the continuous variable.) $\endgroup$ – Scortchi - Reinstate Monica♦ Sep 3 '15 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.