# Sampling from a frequency distribution

I have a dataset representing a frequency distribution:

|item|frequency| |-|-| |a|1| |b|1| |c|1| |d|1| |e|2| |f|2| |g|10| h,143 j,724

The data is fairly exponential in character, with a few items occurring frequently, then a long tail of single occurrences. The dataset is >>1 million lines/items is size. What is the best way to take a random sample from the distribution? This would be python code not R.

## 2 Answers

Python offers many options for sampling from a set with specified frequencies. To give a few widely sued ones:

Native python
Python offers many sueful functions in its random module, specifically

• random.sample() - for sampling without replacement
• random.choices() for sampling with replacement

Numpy
Numpy has its own random module with somewhat different syntax:

• numpy.random.choice() performs sampling with and without replacement, and weith specified probabilities (which may be somewhat inconvenient for the count data, rather than directly sampling from a set, but probably does the same thing).

Pandas
Pandas is a good choice in terms of importinga nd manipulating data, and offers its own sampling function (likely with a numpy backend):

• pandas.DataFrame.sample
• Thanks - very useful as I had not seen the option to pass in both items and frequency lists. I wonder how well it manages very very large lists. Feb 8, 2022 at 15:06

One way you could this is divide your data to groups with equal frequency, then first randomly choose a group such that the probability of choosing a group will be equal to the group frequency (you can do this e.g. by generating a uniform random number $$x$$ between 0 and 1, and selecting group $$i$$ if $$a_i < x < b_i$$ such that $$p_i = b_i - a_i$$)

Then just randomly select one of elements of the group.

• "stratified sampling"? I guess the bin size will be defined by the frequency of the largest item unless I split the largest item (group) into smaller subgroups Feb 8, 2022 at 15:05
• No, my understanding was that you know the frequency of each item, so you don't have to do any binning. But maybe I misunderstood your problem Feb 8, 2022 at 15:19