I have a huge data set that is probably hundreds of millions of rows. This data follows a very skewed power law distribution. Consider the X-axis to be products and the Y-axis to be revenue from products. Almost 95% of revenue would come from 1% of the products. The distribution looks like this:
I want to generate a random sample from this distribution which approximates the original distribution.
All this data is in a huge oracle DB. I see that Oracle SQL has DBMS_RANDOM.VALUE
[link] which generates pseudo-random numbers.
These are my questions:
- What should my sample size be in order to come close to the original data set. Consider original dataset to be 100million rows.
- Doesn't pseudo-random number generators follow a gaussian distribution? If so, isn't it wrong use a random-number generator fit for gaussian distribution over a data which follows a power-law distribution?
- How should I do random sampling over power law distributions? (this can be generalized to random sampling over any custom distribution).