Neural network with missing features in prediction data?

Question

Lets say I want to create a model that predicts an outcome of a certain match. I use features to train the network that are known before a match starts (such as individual performance of each player in the last x months, etc.) and some features that are not known beforehand (just data about how the match went, when certain things happened, etc).

Now, if I want to predict a result of an upcoming match (with some features missing, since they are not know before the match starts). Will my NN perform worse since I used features that I don't have at the time of the prediction to train it or would it perform the same/better compared to a model based on only features that are know beforehand.

And if it's alright to have missing data in the prediction set, how do I go about marking certain features as "missing". Do I simply enter a 0 in those columns?

Answer 1

Your training (and validation) should mimic the prediction environment. That said, the features not available to you at prediction time shouldn't be used at training time. This is the source of your confusion.

Answer 2

I think it is unlikely that the model will perform better than a network trained using only the features known before the start of the match. If all of the other set of features is missing when you make the predictions, there is no way for the network to make use of them.

What you might consider doing is making those features additional outputs of the model. If they are useful attributes, they may represent useful sub-goals for the network, so if it can predict them, it may guide the learning of the hidden layer neurons to provide useful features for predicting the result itself. That way the additional features can be used to give hints in the learning of the main task, without them needing to be available when predictions are made.

Answer 3

While in general it is important not to have training\serving discrepancies, in some cases, it does makes sense to use features that you have only on training time. Sometimes they are called: "calibration features".

Let's take for example some features that are outliers, but affect the result very much and it is only known in hindsight (i.e. during the game). For example: injury of a star player, or whether the game had an overtime or not. You can't (and don't want) to predict these cases but it affects the results during training.

The reason to do that: 1) it makes the training process more efficient (better handling of outliers, faster convergence, better generalization and accuracy). 2) It doesn't affect your prediction at runtime, because at runtime you are not able \ don't want to predict these outliers. In this case, you can use the outlier features on training, but set them to a default (common) value at serving.

Note, that you have to be careful not to overuse it and "snoop" into the data and "cheat" because this will hurt your predictive power.

Another case in which it makes sense to do it, is in ranking systems. Suppose that all you care is to rank (i.e. you don't care about the exact predictions only the ranking). In this case, you can use the feature that is available during training, it improves your training, and then on serving you set a default, same value for all prediction. This should not change the ranking of the final results (it does changes the score but not the rank)

A very common usage for this is in recommendation systems of items and the position feature(where the final position is know only after the ranking) see this article for example.