1
$\begingroup$

I would like to ask a general question which makes me worry when I try to impute NA values.

We know that most of the imputation methods are based on the rest non-NA values. However, if we know that our data set comes from volunteers observations (we don't know if they are trustworthy) also we have a few corrupted measurements ( variables take non-rational values) and also we have let us say a 5% of NA values (these NA values are NOT NA values that correspond to cases that the experiment lets say didn't take place).

So in such cases when the data set is somehow problematic, my opinion is that we should prefer to discard observations (I'm talking about cases where we are allowed to discard observations) in order to introduce as less bias as possible.

In my mind, the imputation methods work like garbage-in, garbage-out.

$\endgroup$

1 Answer 1

1
$\begingroup$

Where an observation is clearly an error for example a study of canary weights had an entry 50 kilograms then delete it but log this in any report of the analysis.

Where you suspect the data may be dubious or corrupt you could say one hot encode these observations as a new variable and model the data with and without the dubious values depending on how you are doing the analysis or modeling.

Again, do not delete or wholly discard data: it may have been very expensive to obtain! Retain it for further investigation and report as such.

$\endgroup$
4
  • $\begingroup$ Are there any paper that discusses the method that you mentioned, in order to take a closer look ?? $\endgroup$
    – Fiodor1234
    Jun 28, 2020 at 18:52
  • $\begingroup$ Hi Fiodor, One hot encoding is very widely used in machine learning, I suggest a Google search if you need details specific to your work. $\endgroup$ Jun 28, 2020 at 19:04
  • $\begingroup$ So as I see it, for example, if we have the variable W, we will extend it to W and W_NA where W_NA will be a vector with 1 on the locations that W had the NA? $\endgroup$
    – Fiodor1234
    Jun 28, 2020 at 19:41
  • 1
    $\begingroup$ Yes I think you have it: if your variable is W and you suspect some of the values are dubious or corrupt then make a new column variable say called W_NA. Fill it with rows so that if the observation was corrupt it is 1 otherwise 0. Good luck! $\endgroup$ Jun 29, 2020 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.