1
$\begingroup$

Suppose I have a dataset with 100 rows, but for one of my columns, titled 'Age', there are NaN values for 14 of the rows. A common approach to dealing with this is filling up those NaN values with the median or mean of the data, but what is the justification for this? I can agree that the median or mean age in this case is the most 'likely' age for a random datapoint if the age histogram looks vaguely Gaussian, but why shouldn't I populate those NaNs with a random number taken from a normal distribution centered at that 'most likely' age? Wouldn't that be more realistic? It seems unrealistic to me that if my dataset is missing 14 people, that they're all going to be the same age, even if it is the most common age. Seems more likely to me there'd be a variance around that most likely age, just like a normal distribution.

$\endgroup$
6
  • 2
    $\begingroup$ what you are saying is perfectly correct. look up multiple imputation eg see stefvanbuuren.name/fimd/sec-simplesolutions.html $\endgroup$
    – seanv507
    Jan 25 at 16:39
  • 2
    $\begingroup$ also be wary of all of the issues that ANY kind of imputation can have on inference if the values are not misssing at random. $\endgroup$
    – bdeonovic
    Jan 25 at 16:43
  • 1
    $\begingroup$ @bdeonovic How can I tell if the values are missing at random or not? Does 'not missing at random' mean, for instance, if all the ages of people from a certain ethnic group are NaN? $\endgroup$
    – sangstar
    Jan 25 at 16:45
  • 1
    $\begingroup$ Yes that example would not be missing at random, but it can of course be much more complicated (some nonlinear relationship between all of your variables, and maybe some unmeasured ones, and the missingness). There are some methods to try to figure that out...but ultimately you can't. This is why high quality data collection and experiment planning is important $\endgroup$
    – bdeonovic
    Jan 25 at 16:49
  • $\begingroup$ We do in fact have a whole tag for this multiple-imputation. I would endorse @seanv507 suggestion that you look at Steff Van Buuren's work though. He is something of an authority in this area. $\endgroup$
    – mdewey
    Jan 25 at 16:57

1 Answer 1

1
$\begingroup$

Well, there is also always the option of removing those rows. You might object, saying you don't want to lose 14/100 rows - but at the same time, you have to consider that the other option is you fill 14/100 rows with fake data. I genuinely don't know which is better, but both are worth considering.

To your question:

I would say that the justification for imputing the mean/median of the data is that by using the mean, your imputation of fake data minimally disturbs the overall distribution you have.

Your suggestion of randomly selecting ages from your existing age distribution is definitely an interesting suggestion, but with only 100 rows and 14 of them needing imputing, think about how much difference there will be in the distribution, from one "random roll of the dice" to the next . With only 100 rows, if you randomly impute, even one extra "rare case" can add a big outlier and significantly affect your data. Yes it's unrealistic for there to be 14 people with all the exact same age (which happens to be at the mean), but that would likely be the most easy way to impute fake data without disrupting the distribution hugely.

Now, what might be an interesting idea to try a random sampling approach but do it many times. So, sample with replacement from your distribution of ages, fill in the NaNs, run your analysis, then repeat (new random sample from your distribution of non-fake ages etc.) . Perhaps you do this a couple dozen times and average the analysis result (depending on what you're trying to do). This might be the best option.

Let's be clear though - missing 14/100 data points is pretty huge, especially for any kind of statistical analysis. You might really be better off just removing them. Please note bdeonovic's comment, though - if they're not missing at random, this could change everything. Everything I've been explaining is assuming they're missing at random.

$\endgroup$
7
  • 2
    $\begingroup$ "Minimally disturbs" does not appear to be generally true: any imputation method that substitutes one common value for all missing data replaces a distribution, which in many cases might be expected to be continuous, with a mixture having a discrete component. For many purposes that's a serious modification! This seems to be the origin of the OP's concern. $\endgroup$
    – whuber
    Jan 25 at 17:06
  • $\begingroup$ @whuber I agree with you, but wouldn't you agree that what I explained is, in fact, the justification used for this kind of imputation? I meant "minimally disturbs" relative to other imputation methods, such as OP's random sampling suggestion. Do you think I should modify my answer to emphasize the potentially large impact of doing so? Is my answer currently misleading? $\endgroup$ Jan 25 at 17:10
  • 2
    $\begingroup$ I don't have any other particular comments to make, because all your advice is reasonable. What I am struggling with is the question: it doesn't really fit our format well because it is vague, overly general, and speculative. It's not asking about a specific problem somebody actually faces. This causes us all to struggle with how to answer it and it makes it difficult to write or even recognize a truly good answer. $\endgroup$
    – whuber
    Jan 25 at 17:15
  • $\begingroup$ My data in this case is not randomly missing, as they overwhelming share the same type of column value (in this case social class), so my best course of action clearly is to remove those rows as getting value out of adding 'synthetic' data just doesn't seem worth the hassle. $\endgroup$
    – sangstar
    Jan 25 at 17:16
  • 1
    $\begingroup$ @sangstar I am inclined to agree, but again it depends on what you are doing. If you're predicting height and you're using age and gender as a predictor variables, and you have age missing for half of the males in your population (that's the non-random part - it's only male ages missing), if you remove those rows and then keeping chugging along, you run into a big issue because now the "gender" variable is skewed too (because now the impact of "male" as gender might be over/under stated since you deleted half of those rows). Think carefully about what you're removing vs. what your task is. $\endgroup$ Jan 25 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.