Variable transformation on Kaggle titanic problem

Question

I was checking Kaggles Titanic problem and a common feature processing is playing with Parch (number of parents) and Sibsp (number of siblings/spouses).

• First they take it as they come.

• Then they mix the two in TotalFamily (Sibsp + Parch)

• Finally they transform the categorical TotalFamily in a binary variable TravelsAlone (TotalFamily = 0)

This seems to improve performance, I'm trying to understand why:

• one variable is better than two? aren't we loosing relations doing this?
• is it because a binary variable is better than a categorical?
• are these good rules of thumb regardless of the algorithm picked (ensembles, svm, etc).

Please avoid "check for yourself and see" answers, as I said, after seeing a lot of people doing the same transformations I think there's a rule of thumb somewhere and I'm trying to find it.

Answer 1

What people are doing is trying different things until they find something that appears to work. There is no reason in particular why transforming a variable that can take many values into a variable that can only take two values should improve matters.

Try to think of it this way. Somewhere out there is data generating process (DGP) that generated the data that you see before you. What you're trying to do is to find a model that allows you to reproduce the data generated by the DGP. If the DGP happens to be a linear then it happens to be linear. If it happens to be non-linear then it happens to be non-linear. If the DGP is a function of a variable that takes many values, then it is a function of variable that takes many values. If the DGP happens to be a function of a variable that takes only two values, then it is a function that only takes two values. It is what it is.

Sometimes they'll find something that appears to reproduce the DGP fairly well only to later find out that it is wrong because it produces different outcomes than the DGP produces. And sometimes it happens to work well. There is no rule of thumb except to guess, check your guess, and then probably guess again.

In this case I imagine some people guessed that it mattered that people were not alone, e.g. if you're alone then there is no one to give your place to and no one to give you a place or to help you get a place. They first tried both separately together, then combined them as they considered that they're a group realized that group size perhaps doesn't matter much, then guessed that the mere fact of being together mattered etc. Later other people noticed or heard that this worked and copied it hence why it's probably so common.

But to answer your question, there is no rule of thumb that I am aware of. There is also no reason why this transformation should work. It's just something that people try when they try to guess what the DGP might look like.

Answer 2

Note that the IBM Watson solution to this problem was utterly wrong, and that the dataset is too small by a factor of 20 for split-sample validation to work properly. The original sources dataset is at http://biostat.mc.vanderbilt.edu/DataSets . Please don't refer to a copy of the original dataset.

sibsp and parch cannot be interpreted correctly without interacting them with age, although there may be some merit in computing a "passenger is alone" variable.

A fully worked out case study using logistic regression for this dataset is in my book Regression Modeling Strategies and its course notes which may be found at the RMS entry under http://www.fharrell.com/p/blog-page.html . In my detailed analysis you'll see that I dealt with the age transformation very thoroughly and flexibly, and properly penalized the analysis for using the data to derive the function forms for age.