1
$\begingroup$

I have a dataset representing a frequency distribution:

|item|frequency| |-|-| |a|1| |b|1| |c|1| |d|1| |e|2| |f|2| |g|10| h,143 j,724

The data is fairly exponential in character, with a few items occurring frequently, then a long tail of single occurrences. The dataset is >>1 million lines/items is size. What is the best way to take a random sample from the distribution? This would be python code not R.

$\endgroup$

2 Answers 2

1
$\begingroup$

Python offers many options for sampling from a set with specified frequencies. To give a few widely sued ones:

Native python
Python offers many sueful functions in its random module, specifically

  • random.sample() - for sampling without replacement
  • random.choices() for sampling with replacement

Numpy
Numpy has its own random module with somewhat different syntax:

  • numpy.random.choice() performs sampling with and without replacement, and weith specified probabilities (which may be somewhat inconvenient for the count data, rather than directly sampling from a set, but probably does the same thing).

Pandas
Pandas is a good choice in terms of importinga nd manipulating data, and offers its own sampling function (likely with a numpy backend):

  • pandas.DataFrame.sample
$\endgroup$
1
  • $\begingroup$ Thanks - very useful as I had not seen the option to pass in both items and frequency lists. I wonder how well it manages very very large lists. $\endgroup$
    – Archie
    Feb 8, 2022 at 15:06
1
$\begingroup$

One way you could this is divide your data to groups with equal frequency, then first randomly choose a group such that the probability of choosing a group will be equal to the group frequency (you can do this e.g. by generating a uniform random number $x$ between 0 and 1, and selecting group $i$ if $a_i < x < b_i$ such that $p_i = b_i - a_i$)

Then just randomly select one of elements of the group.

$\endgroup$
2
  • $\begingroup$ "stratified sampling"? I guess the bin size will be defined by the frequency of the largest item unless I split the largest item (group) into smaller subgroups $\endgroup$
    – Archie
    Feb 8, 2022 at 15:05
  • $\begingroup$ No, my understanding was that you know the frequency of each item, so you don't have to do any binning. But maybe I misunderstood your problem $\endgroup$
    – J. Delaney
    Feb 8, 2022 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.