3
$\begingroup$

I have a dataframe with many (>50%) NAs values and I am looking for a way to deal with it. From what I've found, I think many people recommend using imputation like multiple imputation or using random forest (rf deals with missing values by replacing the NAs with median). However, let me explain my situation with a sub-dataset:

         basket.Africa.1 basket.US.10 basket.France.20

apple           1              0                1

orange         NA              3                2

pear           NA              NA               2

peach           1              NA               NA

banana          1              2                3

Each basket is scan by a machine (pour the fruits from the basket and the machine will scan each fruit), and then the amount of fruits will be recorded. So, there are 1 apple, 1 peach, and 1 banana in basket.Africa.1. Note that basket.Africa.1 refers to basket #1 from Africa.

The NAs here is NOT missing by random, it means it is not possible to have NAs. For example, basket.Africa.1 has NAs in orange and pear, because Africa does not produce any orange and pear, so it is for sure that orange and pear counts are NAs.

In other words, the zero in basket.US.10 (this basket is from the US) does NOT mean missing value, but instead, it is possible to have apple in this basket but there're none in this particular basket (basket # 10 from the US).

My goal is to use the dataset (~100 columns of different baskets from 5 countries (Africa, US, France, China, Australia) and ~10 rows of different fruits), and answer: if I am given a random basket, how can I determine which country is this basket belongs to?

I don't think it is appropriate to fill any values for the NAs because the NAs should have no value! I also try the multiple imputation but my data contains WAY TOO MANY missing values, so this method doesn't work neither...

$\endgroup$

2 Answers 2

2
$\begingroup$

While I can understand why some people would say these data are MNAR (Missing Not At Random), I would rather say that they are missing by design. Another term that is somtimes used is structural missingness.

The way to handle this will depend very much on the model that you fit. One approach is to do nothing and retain the NAs. Some models will impute a value such as the mean or median - and you would not want that, but it might be OK if they ignore/delete NAs. Another approach that may work for you is to use a unique coding for the these data, such as -1 which would otherwise be impossible (since you seem to be observing counts). Obviously that would mean a model that uses a log transformation such as poisson or negative binomial regression would be out of the question, but there should be many others, perhaps tree based, that may work. This is not ideal so I would explore performance using different methods and models.

$\endgroup$
1
$\begingroup$

I don't think it is appropriate to fill any values for the NAs because the NAs should have no value!

What is the reasoning behind this statement? Consider the following example that you've provided:

For example, basket.Africa.1 has NAs in orange and pear, because Africa does not produce any orange and pear, so it is for sure that orange and pear counts are NAs.

Well, if you were to just look at the basket: what are the observed frequencies of pears and oranges in this basked? They are zero - fill the NAs with that.
Your effort to code 'outcome is impossible' with NA might not be necessary, since an event being impossible to occur under given circumstances will almost certainly show in the observed frequencies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.