How to handle non existent (not missing) data?

Question

I've never really found any good text or examples on how to handle 'non-existent' data for inputs to any sort of classifier. I've read a lot on missing data but what can be done about data that cannot or doesn't exist in relation to multivariate inputs. I understand this is a very complex question and will vary depending on training methods used...

Eg if trying to predict laptime for several runners with good accurate data. Amongst many inputs, possible variables amongst many are:

1. Input Variable - First time runner (Y/N)
2. Input Variable - Previous laptime ( 0 - 500 seconds)
3. Input Variable - Age
4. Input Variable - Height . . . many more Input variables etc

& Output Predictor - Predicted Laptime (0 - 500 seconds)

A 'missing variable' for '2.Previous laptime' could be computed several ways but '1. First time runner' would always equal N . But for 'NON EXISTENT DATA' for a first time runner (where '1. First time runner' = Y) what value/treatment should I give for '2. Previous laptime'?

For example assigning '2. Previous laptime' as -99 or 0 can skew the distribution dramatically and make it look like a new runner has performed well.

My current training methods have been using Logistic regression, SVM, NN & Decision trees

Answer 1

Instead of assigning special value for non-existent first time runner previous lap time, simply use interaction term for previous lap time with the inverse of first time runner dummy:

$$Y_i=\beta_0+\beta_1 FTR_i+\beta_2 (NFTR_i)\times PLT_i+...$$

here

• $Y_i$ is your input variable,
• $...$ is your other variables,
• $FTR_i$ is dummy for the first time runner,
• $PLT_i$ is the previous lap time and
• $NFTR_i$ is dummy for non first time runner equaling 1, when $FTR_i=0$ and 0 otherwise.

Then the model for first time runners will be:

$$Y_i=(\beta_0+\beta_1) + ...$$

and for non first time runners:

$$Y_i=\beta_0+ \beta_2 PLT_i + ...$$

Answer 2

For a logistic regression fitted by maximum likelihood, as long as you have both (1) and (2) in the model, then no matter what "default" value that you give new runners for (2), the estimate for (1) will adjust accordingly.

For example, let $X_1$ be the indicator variable for "is a new runner", and $X_2$ be the variable "previous laptime in seconds". Then the linear predictor is:

$\eta = \alpha + \beta_1 X_1 + \beta_2 X_2 + \ldots$

If the default for $X_2$ is zero, then the linear predictor for a new runner is:

$\eta = \alpha + \beta_1 + \ldots$

whereas for an existing runner, it will be:

$\eta = \alpha + \beta_2 X_2 + \ldots$

Now suppose that you change the default for $X_2$ from 0 to -99. Then the linear predictor for a new runner is now:

$\eta = \alpha + \beta'_1 - 99 \beta_2 + \ldots$

but for an existing runner, it will remain the same. So all you've done is reparameterise the model, such that $\beta'_1 - 99 \beta_2 = \beta_1$, and since maximum likelihood is paremeterisation invariant, the estimates will adjust accordingly.

Of course, if you're not using maximum likelihood (i.e. you're using some sort of penalisation or prior on the parameters), then you're going to get different values unless you adjust the penalisation/prior accordingly. And if the model is non-linear (e.g. SVM, NN & Decision trees), then this argument doesn't work at all.