10
$\begingroup$

I'm training a regression NN to generate insurance premiums. I have training data which consists of various metrics about the individuals. The problem is, quite often the dataset is incomplete, for example, I may not know whether the individual smokes, it's just missing data, but for others I do have that data, but in both cases I know what their premium was.

I'd say a considerable portion, maybe 75% is missing at least one item of data across the whole spectrum of what I'll be providing.

I'm not sure how best to handle it? The only thing I could think is if the data is missing then I set it to the zero variance of the normalised data? i.e. it's neutral and has no impact positive or negative?

Any other ideas?

Many thanks

$\endgroup$
11
$\begingroup$

The problem of missing data has in data analysis obtained considerable attention. In their reference book [1] Rubin and Little define three mechanisms behind data becoming missing (definitions from from https://en.wikipedia.org/wiki/Missing_data):

  • MCAR: Values in a data set are missing completely at random (MCAR) if the events that lead to any particular data-item being missing are independent both of observable variables and of unobservable parameters of interest, and occur entirely at random
  • MAR: Missing at random occurs when the missingness is not random, but where missingness can be fully accounted for by variables where there are completely observed
  • MNAR: the value of the variable that's missing is related to the reason it's missing

In the example you give, whether a subject smokes or not, tends often to be missing. I would believe that MNAR is the case for smoking. Non-smokers have no problem with filling in this fact, whereas some (perhaps light) smokers can be reluctant to indicate a 'Yes'. So the missingness of 'Smoking' is most likely to indicate a smoker, but we don't know.

When MNAR is the case, you need to model the missing data mechanism as well. Being creative, it is possible to model a simple missing data mechanism with a neural network. You can represent the boolean variable (like smoker, yes/no) by one input neuron, with encoded input $1$ for smoker and $-1$ for non-smoker. Give the value $0$ as input to this neuron when the smoker variable is missing. Any weights connecting with the 'smoker input neuron' will have no influence on the further computation, because $0 \times w_{i\,j}=0$.

You don't have to adapt the training algorithm or the network topology for this solution to work for boolean and enumerated variables.

[1] Rubin, Donald B.; Little, Roderick J. A. (2002). Statistical analysis with missing data (2nd ed.). New York: Wiley

$\endgroup$
2
  • 3
    $\begingroup$ What about for continuous or numerically valued inputs? $\endgroup$ Oct 10 '19 at 17:44
  • $\begingroup$ @PavelKomarov if you have a new question, it's best to ask it as a new question, not in the comments. $\endgroup$
    – Joooeey
    Feb 23 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.