# Data - how 'smart' are models / is my dummy redundant?

I am currently practicing my data-analysis skills by doing the Titanic competition from kaggle. Its a rather grim exercise where you have a list of passengers and have to predict whether they survived the disaster.

I am planning on using Logistic Regression and am in the process of transforming the data, but I had the following question:

One of the variables (integer) is parch which is described as the # of parents / children aboard the Titanic

I wanted to test the hypothesis that a 'parent' would have lower survival chance, as self preservation was not necessarily their number 1 priority (again, its a grim exercise).

One way I could do this is by adding a parent dummy where (age > 18+ parch > 0) But this would not be new information, as the value of this dummy is based on features/data that already exist in the row.

My question is as follows: Is adding this dummy required? Assuming my hypothesis is true, wouldn't the model infer learn the difference between children and parents itself based on the age variable?

passenger_id | age | parch | parent
1          5      2       0
2          3      2       0
3          40     2       1
4          43     2       1


For example, the dataset above would describe 1 family of 4, would the parent dummy be redundant as the regression already gets both the parch and age variables? How 'capable' are the models in inferring these connections from the data?

The logistic regression model is linear. It can only find an association between age and survival if the relationship is indeed linear. I'm going to explain this in two ways, first mathematically by relying on the definition of the logistic regression model, and second by a simulation study.

# Theory

A linear relationship would for instance be present if a 10 year increase in age means you are twice as likely to die (or equivalently half as likely to survive). In logistic regression, this information is contained in the coefficients, which represent log-odds ratios. Odds are the ratio of two probabilities. Let's consider the odds of survival for a 35 year old individual: $$\text{Odds}(\text{Age}=35) = \frac{P(\text{Passenger survives} \mid \text{Age=35})}{P(\text{Passenger dies} \mid \text{Age=35})} ,$$ where the probabilities are predictions of the logistic regression model for the given variables, here only Age. Next, we consider the odds for an individual who is 10 years older: $$\text{Odds}(\text{Age}=45) = \frac{P(\text{Passenger survives} \mid \text{Age=45})}{P(\text{Passenger dies} \mid \text{Age=45})} .$$ The ratio of both is called odds ratio and denotes the difference in odds of survival between a 45 and 35 year old: $$\frac{P(\text{Passenger survives} \mid \text{Age=45})}{P(\text{Passenger dies} \mid \text{Age=45})} / \frac{P(\text{Passenger survives} \mid \text{Age=35})}{P(\text{Passenger dies} \mid \text{Age=35})} .$$ Referring to the example from above where a 10 year increase in age halves the chance of survival, the odds ratio would be $1/2$. The coefficient for Age in a fitted logistic regression model is closely related to this quantity: $$\exp( 10 \cdot \beta_\text{Age}) = \frac{P(\text{Passenger survives} \mid \text{Age=45})}{P(\text{Passenger dies} \mid \text{Age=45})} / \frac{P(\text{Passenger survives} \mid \text{Age=35})}{P(\text{Passenger dies} \mid \text{Age=35})} ,$$ where the factor 10 comes from the fact that coefficients consider a one unit increase, thus we have to scale the coefficient accordingly. If the assumption of a linear relationship between survival and age holds, we should obtain $$\exp( 10 \cdot \beta_\text{Age}) = \frac{1}{2} \Leftrightarrow \beta_\text{Age} = 0.1 \cdot \log \frac{1}{2} \approx −0.069 .$$ Because the predicted probability of survival (death) given age is defined as \begin{align} P(\text{Passenger survives} \mid x_\text{Age}) &= \frac{\exp(\beta_0 + \beta_\text{Age} \cdot x_\text{Age})}{1 + \exp(\beta_0 + \beta_\text{Age} \cdot x_\text{Age})} , \\ P(\text{Passenger dies} \mid x_\text{Age}) &= 1 - P(\text{Passenger survives} \mid x_\text{Age}) , \end{align} we can derive that the logistic regression model is linear with respect the log-odds ratio: $$\log\left( \frac{P(\text{Passenger survives} \mid x_\text{Age})}{P(\text{Passenger dies} \mid x_\text{Age})} \right) = \beta_0 + \beta_\text{Age} \cdot x_\text{Age} ,$$ where $\beta_0$ is a bias (intercept) term.

Most importantly, $\beta_\text{Age}$ is a constant, which always acts the same on age. In your example, you make a hard-threshold and assume that children do much better than adults, which would correspond to $\beta_\text{Age}$ promoting survival more for individuals of age less than 18 than for those older than 18. This clearly violates the result from above that the log-odds ratio is a constant for all ages. In conclusion, a standard logistic regression model cannot detect this effect accurately.

By dichotomising age, you can actually model the desired effect because the odds ratio has the form $$\frac{P(\text{Passenger survives} \mid \text{Age < 18})}{P(\text{Passenger dies} \mid \text{Age < 18})} / \frac{P(\text{Passenger survives} \mid \text{Age \geq 18})}{P(\text{Passenger dies} \mid \text{Age \geq 18})} = \exp(\beta_\text{Age<18})$$ Hence, feature engineering does help a lot.

# Simulation

To demonstrate what the impact on prediction performance is, I carried out a simulation study. The dataset has two variables: gender and age. The coefficients have been chosen such that the odds ratio for survival with respect to females vs males is $3/2 = 1.5$ and odds ratio for Age $<$ 18 vs Age $\geq$ 18 is 4. Below is the distribution of survival for the different age groups:

Overall distribution of outcome:

 dead alive
124    76


Distribution of outcome among children ($<18$):

 dead alive
2    25


Distribution of outcome among adults ($\geq 18$):

 dead alive
122    51


Fitting a simple linear logistic regression model and predicting survival on an independent test set of 200 patients, I obtained a ROC AUC value of 0.752 (1.0 is optimal). The corresponding model is

             Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.541338   0.409447  -1.322 0.186129
age         -0.027525   0.007904  -3.482 0.000497 ***
sexfemale    2.171798   0.366276   5.929 3.04e-09 ***


The model associated increasing age with a slightly decreasing chance of survival. The model is on the right path, but underestimates the impact of age.

Doing the same for a model that in addition contains a dichotomised age variable, I obtained ROC AUC value of 0.796, which is quite an improvement. The corresponding model is

                 Estimate Std. Error z value Pr(>|z|)
(Intercept)     -3.225066   0.719255  -4.484 7.33e-06 ***
I(age < 18)TRUE  4.838264   0.974587   4.964 6.89e-07 ***
age              0.008502   0.010303   0.825    0.409
sexfemale        2.834142   0.505019   5.612 2.00e-08 ***


In this model, it is obvious that age does not have a linear effect on survival, in fact, the coefficient for age is not significant any more. The dichotomised variable on the other hand is highly significant and has a large odds ratio. This demonstrates that the feature engineering can make or break a logistic regression model.

The code for the simulation study is available at https://gist.github.com/sebp/2b5bf8e9f323cce8454f72d5365abf9f

With a logistic regression, where you only put age and parch into the model as linear predictors, you will have the predicted probability of survival monotonically increasing or decreasing (or, of course, constant) in each variable. In fact, it would act linearly on the logit of the survival probability. Your assumption is that for passenger $i=1,2,\ldots$ you have $$\log \pi_i - \log(1-\pi_i) = \beta_0 + \beta_1 \times \text{age} + \beta_2 \times \text{parch},$$ where $\pi_i = P(\text{passenger } i \text{ survives})$. So, it would not be able to reflect it, if survival probability keeps going up the more age with increasing age, until you become elderly and a bit frail. It will just try to do a linear line on the logit scale. Within the logistic regression framework you can become more flexible in this respect by using regression splines to allow for non-linear relationships (and interactions with splines to address some of the other issues mentioned below).

Even if it is a linear line on that scale, unless you are a parent, it would not be able to reflect that. A change in the age slope e.g. based on parch would be possible with an age by parch interaction, but again, if your belief is that the effect of parch changes exactly at age 18 years, then you probably actually want and interaction with the indicator for whether a passengers age is $\geq 18$, otherwise you try to approximate that step-function by a linear line (probably not the best idea).

Picking/constructing variables/predictors that you believe will work well, or that you want to test out is "feature engineering". Algorithms like logistic regression take what you give them and the assumptions you (implicitly make such as linear relationships on some scale) and then do the best (with respect to the loss function) they can with that. If you did build really clever features based on your deep insights into shipping disasters, logistic regression may perform fantastically, if you give it terrible features, it will do poorly.

If you want a model that figures this out on its own (at least given enough data), then you may want something like extreme gradient boosting (xgboost), random forest or deep neural network ("deep learning"). These can to some extent construct their own features/figure out what interactions occur may matter. However, they need a lot of data to do so. So, if you give them good features, they will need a lot less training data to achieve the same performance as you would get with poorly designed input features. Given an near infinite amount of data, your feature engineering will often be irrelevant, for small datasets like the Titanic one it will still matter a lot.