1
$\begingroup$

I am wondering how to deal with a variable having what I call natural NaN values.

For example, a measure of duration between 2 events. If one event did not occure the variable has no value. For that variable, I cannot put 0 or the mean or what ever...

Did you meet that case and how did you proceed ?

Many algorithms do not accept NaN values so one way or another it has to be dealt with.

More specifics about the case : my purpose is to detect claim insurance fraud. It is a classification problem. I describe what is a fraudulent claim with many variables, some of them have natural NaN values. For example, the time between a claim and a past amendment to the contract is such a variable. If there was no amendment prior to the claim that variable is NaN.

Please note that the time between a past event ( contract amendment ) and the claim will never have any value because the amendment has not happened ( before the claim ). Humans in charge of fraud detection use that kind of information for their investigations so I guess I must use it as well.

Improve this question
$\endgroup$
5
  • $\begingroup$ If no events occurred, then you can't estimate the time between events. Skip that item. Maybe another will have some useful data. $\endgroup$
    – BruceET
    Commented Apr 2, 2019 at 20:11
  • 1
    $\begingroup$ It is less about ideas, and more about the specifics of your research question, choice of statistical test/model, and data. If, for example, you are were doing a survival (time-to-event) model and this variable was your dependent variable, then you would typically assign a value based on time the subject was studied and assign a censoring flag to that subject (in a separate variable) to indicate that the event of interest did not occur for that subject before the end of the study period. So I recommend that you provide more specifics about your case. $\endgroup$
    – AlexK
    Commented Apr 3, 2019 at 0:26
  • 1
    $\begingroup$ There was a similar question here with an answer that discusses handling of right-censored predictors. $\endgroup$
    – AlexK
    Commented Apr 3, 2019 at 7:33
  • $\begingroup$ Another similar question: stats.stackexchange.com/q/56306/241093 $\endgroup$
    – AlexK
    Commented Apr 3, 2019 at 7:37
  • $\begingroup$ @Fabrice BOUCHAREL: Can you please add all new information in comments as an edit to the original Q, so that all information is in one place. That way the probability of a useful answer will increase. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Apr 3, 2019 at 11:56

1 Answer 1

Reset to default
1
$\begingroup$

In your example, I would not impute the missing value. The data aren't "really" missing, they just haven't been generated yet. That would be like me imputing how much money you are going to spend before you spend it. Imputing missing values might have the effect of biasing your results, which would be very bad. If there are only a few missing, I would just remove them. Those observations haven't contributed the information you want into the model. If they make up the bulk of your data, I would rethihnk your approach.

If you absolutely must include them, use an implementation which allows for NaNs (like LightGBM).

Improve this answer
$\endgroup$
1
  • $\begingroup$ The time between a past event ( contract amendment ) and the claim will never have any value because the amendment has not happened ( before the claim ). Humans in charge of fraud detection use that kind of information for their investigations so I guess I must use it as well. $\endgroup$
    – Fabrice BOUCHAREL
    Commented Apr 4, 2019 at 6:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.