0
$\begingroup$

I'm looking for an approach that would fill null values in a dataframe for discrete and continuous values such that the nulls would be replaced by randomly generated numbers that reflect the relative frequency of the values in the column. It would looks something like this for a categorical variable:

IN: dataframe['fruit']
OUT:
apple
banana
apple
apple
nan
apple

Given this data, it would fill the nan with apple approximately 4/5 of the time and banana 1/5 of the time

IN: dataframe['fruit'].fillna('randomwalk')
OUT:
apple
banana
apple
apple
apple
apple
$\endgroup$
  • 2
    $\begingroup$ Software-specific questions are arguably off-topic here. It seems that @Dougal has saved this thread by answering in fairly general terms and bringing out the underlying statistical question, but please see advice in the Help Center for your future questions. $\endgroup$ – Nick Cox Dec 3 '15 at 9:33
1
$\begingroup$

I wouldn't call this a random walk, just random sampling.

The algorithm is pretty simple:

  1. Get the frequencies for each column, probably with value_counts.
  2. Divide by the number of nonnull points to get a distribution.
  3. Sample from that distribution a number of times equal to the number of null items to fill. np.random.choice can do that easily; give the weights as the distribution above.
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.