0
$\begingroup$

I'm looking for an approach that would fill null values in a dataframe for discrete and continuous values such that the nulls would be replaced by randomly generated numbers that reflect the relative frequency of the values in the column. It would looks something like this for a categorical variable:

IN: dataframe['fruit']
OUT:
apple
banana
apple
apple
nan
apple

Given this data, it would fill the nan with apple approximately 4/5 of the time and banana 1/5 of the time

IN: dataframe['fruit'].fillna('randomwalk')
OUT:
apple
banana
apple
apple
apple
apple
$\endgroup$

closed as off-topic by Sycorax, Greenparker, Dimitriy V. Masterov, COOLSerdash, gung Jul 28 '16 at 11:53

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Sycorax, Greenparker, Dimitriy V. Masterov, COOLSerdash, gung
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    $\begingroup$ Software-specific questions are arguably off-topic here. It seems that @Dougal has saved this thread by answering in fairly general terms and bringing out the underlying statistical question, but please see advice in the Help Center for your future questions. $\endgroup$ – Nick Cox Dec 3 '15 at 9:33
1
$\begingroup$

I wouldn't call this a random walk, just random sampling.

The algorithm is pretty simple:

  1. Get the frequencies for each column, probably with value_counts.
  2. Divide by the number of nonnull points to get a distribution.
  3. Sample from that distribution a number of times equal to the number of null items to fill. np.random.choice can do that easily; give the weights as the distribution above.
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.