When there are NaN values for a column of data, why is it okay to fill the values with the median or mean of that column? Suppose I have a dataset with 100 rows, but for one of my columns, titled 'Age', there are NaN values for 14 of the rows. A common approach to dealing with this is filling up those NaN values with the median or mean of the data, but what is the justification for this? I can agree that the median or mean age in this case is the most 'likely' age for a random datapoint if the age histogram looks vaguely Gaussian, but why shouldn't I populate those NaNs with a random number taken from a normal distribution centered at that 'most likely' age? Wouldn't that be more realistic? It seems unrealistic to me that if my dataset is missing 14 people, that they're all going to be the same age, even if it is the most common age. Seems more likely to me there'd be a variance around that most likely age, just like a normal distribution.
 A: Well, there is also always the option of removing those rows. You might object, saying you don't want to lose 14/100 rows - but at the same time, you have to consider that the other option is you fill 14/100 rows with fake data. I genuinely don't know which is better, but both are worth considering.
To your question:
I would say that the justification for imputing the mean/median of the data is that by using the mean, your imputation of fake data minimally disturbs the overall distribution you have.
Your suggestion of randomly selecting ages from your existing age distribution is definitely an interesting suggestion, but with only 100 rows and 14 of them needing imputing, think about how much difference there will be in the distribution, from one "random roll of the dice" to the next . With only 100 rows, if you randomly impute, even one extra "rare case" can add a big outlier and significantly affect your data. Yes it's unrealistic for there to be 14 people with all the exact same age (which happens to be at the mean), but that would likely be the most easy way to impute fake data without disrupting the distribution hugely.
Now, what might be an interesting idea to try a random sampling approach but do it many times. So, sample with replacement from your distribution of ages, fill in the NaNs, run your analysis, then repeat (new random sample from your distribution of non-fake ages etc.) . Perhaps you do this a couple dozen times and average the analysis result (depending on what you're trying to do). This might be the best option.
Let's be clear though - missing 14/100 data points is pretty huge, especially for any kind of statistical analysis. You might really be better off just removing them. Please note bdeonovic's comment, though - if they're not missing at random, this could change everything. Everything I've been explaining is assuming they're missing at random.
