Questions tagged [zipf]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
5 votes
1 answer
314 views

If my data doesn't completely follow the Zipf's law, how do I justify it mathematically?

Zipf's law states that in a text set $s=1$ a few words occur very often, and many words hardly ever occur. Zipf’s law for text sets $s = 1$ in the Zipf distribution defined by: $$f(k; s, N) = \frac{k^{...
user avatar
0 votes
0 answers
12 views

Determining Frequency for Rankings based on their timeline

I have a dataset that reflects rankings for products over a long time period. I do not know how the ranking represents sales, but my first assumption is that I can apply Zipf's Law. However, I don't ...
user avatar
  • 111
0 votes
1 answer
31 views

How do I calculate a proper sample size to get a suitable power of Kolmogorov-Smirnov test with an underlying Zipf distribution?

I have been trying to develop to calculate the sample size to maximize the power of KS test (±0.8) on an underlying Zipf distribution. I have tried estimating the power by performing simulations: <...
user avatar
2 votes
0 answers
56 views

What is a dragon king?

I am studying a system of cities where the largest city appears to be in many aspects an outlier. The distribution of city size - in any country - are often claimed to follow Zipf's law. According to ...
user avatar
1 vote
0 answers
23 views

Zipf's Law: Estimating # words over N [closed]

Say I have a large corpus of p words, and constant (f.r = C) equals p.(1/10). How do you go ...
user avatar
  • 111
0 votes
1 answer
105 views

Interpreting KL Divergence

I'm trying to compare different approaches to rank predictions. I have the ground truth distribution $P$ (discrete, zeta distribution) and two or more distributions ($Q, Q', Q'', Q'''$ in this case) I'...
user avatar
3 votes
1 answer
804 views

Discrete Pareto Distribution vs Zipf Distribution and Power Law vs Zipf Law

I need to get a simple, but clear idea of Discrete Pareto Distribution vs Zipf Distribution and Power Law vs Zipf Law. (Are they similar/ how they relate to each other.) Wikipedia definitions do not ...
user avatar
0 votes
1 answer
216 views

need explanation about the exponent parameter s in zipf distribution

I need to model the popularity of some requested files from a library with Zipf distribution and I want to simulate it in MATLAB. I don't know what's the effect of parameter s on my result. for ...
user avatar
  • 25
1 vote
0 answers
606 views

Practically speaking, is the TF-IDF threshold universal across different corpus?

I would like to know the practical threshold of the TF-IDF (just like the practical p-value cutoff of 0.1 or 0.05 in hypothesis tests). I tried to look at it in some previous post, and some people ...
user avatar
  • 133
10 votes
2 answers
675 views

Is KS test really appropriate when validating a power law/estimating power law parameters?

I'm attempting to find out whether some highly skewed data are drawn from a power law distribution, following the popular paper by Clauset, Shalizi and Newman, 2009. Clauset et al. use the Kolmogorov-...
user avatar
  • 103
2 votes
1 answer
876 views

Calculate Zipf-Mandelbrot parameters from distribution

I am fetching trending topics from social media where the frequency of likes is said to follow a Zipf-Mandelbrot distribution; i.e., some of the posts will have a high number of likes and some other ...
user avatar
5 votes
2 answers
2k views

Characterizing/Fitting Word Count Data into Zipf / Power Law / LogNormal

Using NLTK and Pandas, I was able to process some text files and generate word count data for them, and finally create a histogram describing word frequency. However, I'm wondering what kind of ...
user avatar
0 votes
0 answers
109 views

What is a hooked powerlaw?

I happened to encounter lots of scientific/business scenarios where a Zipf/Pareto/powerlaw describes well my data. However, whenever the mean of the distribution is large enough, the fact that these ...
user avatar
  • 839
1 vote
0 answers
118 views

Rank correlation over time for Zipf-like distribution

I've got 14 years of city population data. I wish to get an idea of how much the ranking of cities by population size changes over time. Is it stable or unstable? I guess if there's not much ...
user avatar
  • 235
4 votes
1 answer
2k views

How to determine if Zipf's Law can be applied?

I'm trying to use feature hashing (or hashing trick) on a set of files that are composed mostly in English and code (in English). I'm trying to see if Zipf's law applies to the set of files before I ...
user avatar
5 votes
1 answer
202 views

Besides the Pareto and Zipfian distributions, which distributions obey the power-law?

I need a list of distributions that obey the power-law, beside the commonly used Pareto and Zipfian distributions. A comprehensive list or a reference to a comprehensive list will be particularly ...
user avatar
3 votes
1 answer
481 views

Fitting Zipf Mandelbrot and use Chi-square test in R

I have a dataset with hashtags and their frequencies (~370k frequencies), as for example (after a sort): ...
user avatar
  • 41
3 votes
2 answers
647 views

How to asses the optimal bag of words vector size?

I have a corpus with 6040592 words and 309074 types (different words). Knowing this information it is possible to know the optimal size of bag of words vectors in order to represent phrases? I am ...
user avatar
  • 131
2 votes
0 answers
71 views

Distinct items in a Zipfian distribution

I recently generated synthetic data sets with presumably Zipfian distribution and noticed that the frequency of the ranks doesn't change whether I generate 10^3 items or 10^6 items or a different ...
user avatar
  • 31
0 votes
1 answer
60 views

Are Power Laws closed under monotonic transformation?

Let $X$ be a random variable that obeys power law. Let $m(\cdot)$ be a monotonic function. Does $Y=m(X)$ also obey power law? Can we then extend this to say that if $X$ follows Zipf's law, $m(X)$ also ...
user avatar
  • 1
8 votes
2 answers
4k views

How to verify if data follows Zipf's law without looking at the graph

I want to check if a given text sample was written by real people or not so I think Zipf's law could help. If data follows Zipfian distribution, the most frequent word will occur approximately twice ...
user avatar
  • 181
2 votes
0 answers
635 views

Understanding Zipf's law calculation for probability of a word

So, I am analyzing a a corpus of ~1,000,000 words and I am trying to understand how I can use Zipf's law. I found on some textbook slides the following formulas: $$f_r = c/r$$ $$p_r = k/r $$ where $...
user avatar
2 votes
0 answers
154 views

Exponent value in long tail phenomenon of online store

I've read (pdf) that there is a long-tail phenomenon (or generalized Zipf law) in popularity of retail items: If I sort items for sale by their popularity, I will see a long tail of not very popular ...
user avatar
  • 409
1 vote
0 answers
38 views

Estimating distribution of traffic across all Web pages

The web is estimated to have approx 5 billion web pages ( http://www.worldwidewebsize.com/ ). How would you go about estimating a CDF to know how many pages are required that make up, say, 90% of ...
user avatar
8 votes
1 answer
4k views

Connection between power law and Zipf's law

I am trying to better understand the connection between the power law distribution and Zipf's distribution (law). There is a neat explanation in [1]. The article suggests that as we can derivate the ...
user avatar
  • 1,114
1 vote
0 answers
533 views

Using Zipf's law to select top K ngrams?

Lets say we extract all the ngrams (unigrams, bigrams, trigrams) along with their frequency from a text. Now if we want to select say some Top K ngrams, is there a way to estimate a good value of K ...
user avatar
5 votes
0 answers
117 views

Zipf's vs Self-similar: are they really the same

Recently I ran into a test using both zipf's and self-similar generated datasets. I followed the description from Jim Gray's paper on generating such datasets (Quickly Generating Billon-Record ...
user avatar
2 votes
0 answers
335 views

Sales prediction of Amazon Sales Rank

I understand that Zipf's Rank-size distribution would allow me to estimate the sales of a book given the amazon sales rank. How can I fit the distribution without knowing book sales leading to a rank. ...
user avatar
1 vote
1 answer
873 views

How to calculate zipf exponent of page degree distribution?

I want to calculate the exponent of the function which shows zipf's law in data. I'm working in the context of graph node degree analysis and want to comprehend results shown by Border et al. on a new ...
user avatar
  • 121
2 votes
1 answer
378 views

Discretize frequency of words that follow zipfian distribution

How could I discretize the frequency of words found in a corpus that follow a zipfian distribution? Are there standard methods? It should create bins of exponential-increasing size. My goal is to ...
user avatar
3 votes
0 answers
554 views

Mean and median in Zipf's distribution

I have a collection of $10^5$ essays, each of which on average contains $10^3$ distinct words. There are $10^6$ distinct words in the entire collection. If I index every word what is the mean and ...
user avatar
  • 375
12 votes
3 answers
5k views

How to estimate parameters for Zipf truncated distribution from a data sample?

I have a problem with the estimation parameter for Zipf. My situation is the following: I have a sample set (measured from an experiment that generates calls that should follow a Zipf distribution). ...
user avatar
  • 265
28 votes
7 answers
19k views

How to calculate Zipf's law coefficient from a set of top frequencies?

I have several query frequencies, and I need to estimate the coefficient of Zipf's law. These are the top frequencies: ...
user avatar
  • 299
7 votes
3 answers
5k views

Is principal components analysis valid if the distribution(s) are Zipf like? What would be similar to PCA but suited to non gaussian data?

I'm analyzing people based on their twitter stream. We are using a 'word bag' model of users, which basically amounts to counting how often each word appears in a persons twitter stream (and then ...
user avatar
  • 173