Engineering features that depend on more than one data point (classification with gradient boosting in particular)?

Question

So I'm working on the Titanic data set (predicting survival of passengers), and would like to add a feature that indicates whether a given passenger's family survived or not (using the known training data). The problem I'm having is when to add the feature.

Naturally I don't want the training set to have knowledge of the test set. However, if I use a CV grid search on the training set after creating the new feature, any split of this training set will create two sets that contain information relating to each other.

Is this a problem (I think it is!)? If so, how do I fix it? I was thinking of using the pipeline function but I'm not exactly sure how to implement it.

Answer 1

It sounds like you are using sklearn, in which case I can confirm that a pipeline will be able to accomplish what you describe. Cross Validated isn't the right forum specifically for programming related questions, so forgive me if I don't walk you through an exact solution. However, this is a statistical analysis forum, so you'll allow me to share some thoughts on your proposed methodology.

I think you must acknowledge the assumptions you're making and determine whether they would hold in a real application. What you assume is that when predicting the response of a given individual, you will have information on the responses of his family. In the case of the titanic, the deaths of all individuals are broadly contemporaneous. So it seems unlikely you would have information on the status of an individual's family but not on the individual. If you create a model on training data where you do have this unusual information, you will find that you cannot apply the model to future data where you don't have this information. In the case of the titanic disaster, it might be silly to think of predicting a larger "population" since we already have data on everyone on the ship, and there is no larger population. But, if this is typical of Kaggle competitions, it is not typical of real applications. Consider, for example, you want to use your model to predict the relative risk of death of individuals on another similar ship in the case of a disaster. Your model clearly would not be applicable since it would not have, in advance, any information on the outcomes of a person's family.

Because many people use Kaggle competitions to practice modeling skills, it is crucial to question the methodologies and avoid exaggerating performance through data leakage, dredging, and overfitting.