4
$\begingroup$

I know this has been asked in the past many times, but i could not find an adequate answer to my problem. I have a dataset with many NaN values. I am making the calculated assumption that these values were not filled in purpose. Probably 50% of observations in continuous columns have a NaN value.

So I ask you:

Is it a good idea to replace all NaN values to -999? I am not planning on running a parametrical model so i suppose the -999 value will not really hurt my model.

On the contrary, i believe that by replacing with -999, i can find a possible pattern between observations that have a value, and the ones who do not.

Is my line of thinking correct?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
8
$\begingroup$

Filling in -999 can and almost certainly will hurt you no matter whether you use a parametric model or not. To be honest, I don't understand your argument when you write that

I am not planning on running a parametrical model so i suppose the -999 value will not really hurt my model.

Far better to read up on . Learn about the different kinds of missingness, e.g., Missing At Random (MAR), Missing Completely At Random (MCAR) etc.

If one predictor has many entries missing, you could simply remove the predictor, since it will likely not contribute any information, anyway. For missing values that remain, you can think about row-wise deletion, possibly imputation.

If missing values are as prevalent in your data set as you write, your treatment of them will probably heavily impact your results. So I strongly recommend that you think long and hard about what you will do, and if at all possible try multiple approaches and see whether the results are consistent. (And don't just fill in -999.)

Related: Why do some people use -999 or -9999 to replace missing values?

Improve this answer
$\endgroup$
2
  • $\begingroup$ What i meant with "I am not planning on running a parametrical model so i suppose the -999 value will not really hurt my model." is that i will be mostly using decision trees. If i used a regression, the -999 would have a huge impact but with a decision tree, i am inclined to believe the splits wont be affected that much or they will at least be affected positively. $\endgroup$
    – Toutsos
    Nov 10, 2017 at 9:42
  • $\begingroup$ Why shouldn't your decision trees split spuriously on this predictor, especially if missingness is possibly not at random? $\endgroup$
    – Stephan Kolassa
    Nov 10, 2017 at 9:52
2
$\begingroup$

What kind of pattern do you expect to see? If you search for a pattern on state missing/non-missing then why not build a binary variable with 0/1?

If you want to see a visualization, then replacing NaNs with something really different then usual values, your replacement is one solution, but not the best.

I do not advise you to do this sort of replacements because most of the scientific software is able to handle missing values as NaNs and if you replace it you blow out almost any kind of sample statistic you can compute.

And finally, the main reason why this sort of not NaN values are used as replacements for NaNs is due to software limitation, most of the time. After 20 year of production code, one of the things I learned is to avoid to take a decision which can influence your way of thinking just because of a software or hardware difficulty.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for this answer. The only reason i do not want to replace with 0/1 is because i expect to also be a pattern between the different values of the observations. AKA, i expect 3 different outcomes depending on if the continuous variable is missing, high or low. I suppose that i could still just make a categorical variable out of this but i would suspect a model would find the best "cutting" points. $\endgroup$
    – Toutsos
    Nov 10, 2017 at 7:10
0
$\begingroup$

Is it a good idea to replace all NaN values to -999? I am not planning on running a parametrical model so i suppose the -999 value will not really hurt my model.

Maybe. Why don’t you just test whether the choice of sentinel value (e.g. -999, -9999 or +99999) has any noticeable impact on your results?

On the contrary, i believe that by replacing with -999, i can find a possible pattern between observations that have a value, and the ones who do not.

Yes it is possible. Alternatively you could replace NaNs with zeroes or mean value of the particular predictor and add one additional binary indicator predictor (0 where value available, 1 for NaN).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.