Questions tagged [zipf]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Ranked frequency distribution table — is same rank assigned for same frequencies?

Let's say I have a word frequency table with rankings, which I will be using to plot a Zipf distribution: ...
0
votes
0answers
29 views

Interpreting KL Divergence

I'm trying to compare different approaches to rank predictions. I have the ground truth distribution $P$ (discrete, zeta distribution) and two or more distributions ($Q, Q', Q'', Q'''$ in this case) I'...
2
votes
1answer
254 views

Discrete Pareto Distribution vs Zipf Distribution and Power Law vs Zipf Law

I need to get a simple, but clear idea of Discrete Pareto Distribution vs Zipf Distribution and Power Law vs Zipf Law. (Are they similar/ how they relate to each other.) Wikipedia definitions do not ...
0
votes
1answer
82 views

need explanation about the exponent parameter s in zipf distribution

I need to model the popularity of some requested files from a library with Zipf distribution and I want to simulate it in MATLAB. I don't know what's the effect of parameter s on my result. for ...
1
vote
0answers
412 views

Practically speaking, is the TF-IDF threshold universal across different corpus?

I would like to know the practical threshold of the TF-IDF (just like the practical p-value cutoff of 0.1 or 0.05 in hypothesis tests). I tried to look at it in some previous post, and some people ...
3
votes
0answers
279 views

Is KS test really appropriate when validating a power law/estimating power law parameters?

I'm attempting to find out whether some highly skewed data are drawn from a power law distribution, following the popular paper by Clauset, Shalizi and Newman, 2009. Clauset et al. use the Kolmogorov-...
2
votes
1answer
517 views

Calculate Zipf-Mandelbrot parameters from distribution

I am fetching trending topics from social media where the frequency of likes is said to follow a Zipf-Mandelbrot distribution; i.e., some of the posts will have a high number of likes and some other ...
4
votes
2answers
1k views

Characterizing/Fitting Word Count Data into Zipf / Power Law / LogNormal

Using NLTK and Pandas, I was able to process some text files and generate word count data for them, and finally create a histogram describing word frequency. However, I'm wondering what kind of ...
0
votes
0answers
78 views

What is a hooked powerlaw?

I happened to encounter lots of scientific/business scenarios where a Zipf/Pareto/powerlaw describes well my data. However, whenever the mean of the distribution is large enough, the fact that these ...
1
vote
0answers
117 views

Rank correlation over time for Zipf-like distribution

I've got 14 years of city population data. I wish to get an idea of how much the ranking of cities by population size changes over time. Is it stable or unstable? I guess if there's not much ...
3
votes
1answer
1k views

How to determine if Zipf's Law can be applied?

I'm trying to use feature hashing (or hashing trick) on a set of files that are composed mostly in English and code (in English). I'm trying to see if Zipf's law applies to the set of files before I ...
5
votes
1answer
174 views

Besides the Pareto and Zipfian distributions, which distributions obey the power-law?

I need a list of distributions that obey the power-law, beside the commonly used Pareto and Zipfian distributions. A comprehensive list or a reference to a comprehensive list will be particularly ...
3
votes
1answer
395 views

Fitting Zipf Mandelbrot and use Chi-square test in R

I have a dataset with hashtags and their frequencies (~370k frequencies), as for example (after a sort): ...
1
vote
1answer
378 views

How to asses the optimal bag of words vector size?

I have a corpus with 6040592 words and 309074 types (different words). Knowing this information it is possible to know the optimal size of bag of words vectors in order to represent phrases? I am ...
2
votes
0answers
61 views

Distinct items in a Zipfian distribution

I recently generated synthetic data sets with presumably Zipfian distribution and noticed that the frequency of the ranks doesn't change whether I generate 10^3 items or 10^6 items or a different ...
0
votes
1answer
46 views

Are Power Laws closed under monotonic transformation?

Let $X$ be a random variable that obeys power law. Let $m(\cdot)$ be a monotonic function. Does $Y=m(X)$ also obey power law? Can we then extend this to say that if $X$ follows Zipf's law, $m(X)$ also ...
6
votes
2answers
3k views

How to verify if data follows Zipf's law without looking at the graph

I want to check if a given text sample was written by real people or not so I think Zipf's law could help. If data follows Zipfian distribution, the most frequent word will occur approximately twice ...
2
votes
0answers
564 views

Understanding Zipf's law calculation for probability of a word

So, I am analyzing a a corpus of ~1,000,000 words and I am trying to understand how I can use Zipf's law. I found on some textbook slides the following formulas: $$f_r = c/r$$ $$p_r = k/r $$ where $...
2
votes
0answers
152 views

Exponent value in long tail phenomenon of online store

I've read (pdf) that there is a long-tail phenomenon (or generalized Zipf law) in popularity of retail items: If I sort items for sale by their popularity, I will see a long tail of not very popular ...
1
vote
0answers
36 views

Estimating distribution of traffic across all Web pages

The web is estimated to have approx 5 billion web pages ( http://www.worldwidewebsize.com/ ). How would you go about estimating a CDF to know how many pages are required that make up, say, 90% of ...
6
votes
1answer
3k views

Connection between power law and Zipf's law

I am trying to better understand the connection between the power law distribution and Zipf's distribution (law). There is a neat explanation in [1]. The article suggests that as we can derivate the ...
1
vote
0answers
465 views

Using Zipf's law to select top K ngrams?

Lets say we extract all the ngrams (unigrams, bigrams, trigrams) along with their frequency from a text. Now if we want to select say some Top K ngrams, is there a way to estimate a good value of K ...
5
votes
0answers
111 views

Zipf's vs Self-similar: are they really the same

Recently I ran into a test using both zipf's and self-similar generated datasets. I followed the description from Jim Gray's paper on generating such datasets (Quickly Generating Billon-Record ...
2
votes
0answers
325 views

Sales prediction of Amazon Sales Rank

I understand that Zipf's Rank-size distribution would allow me to estimate the sales of a book given the amazon sales rank. How can I fit the distribution without knowing book sales leading to a rank. ...
1
vote
1answer
786 views

How to calculate zipf exponent of page degree distribution?

I want to calculate the exponent of the function which shows zipf's law in data. I'm working in the context of graph node degree analysis and want to comprehend results shown by Border et al. on a new ...
2
votes
1answer
363 views

Discretize frequency of words that follow zipfian distribution

How could I discretize the frequency of words found in a corpus that follow a zipfian distribution? Are there standard methods? It should create bins of exponential-increasing size. My goal is to ...
2
votes
0answers
500 views

Mean and median in Zipf's distribution

I have a collection of $10^5$ essays, each of which on average contains $10^3$ distinct words. There are $10^6$ distinct words in the entire collection. If I index every word what is the mean and ...
10
votes
3answers
4k views

How to estimate parameters for Zipf truncated distribution from a data sample?

I have a problem with the estimation parameter for Zipf. My situation is the following: I have a sample set (measured from an experiment that generates calls that should follow a Zipf distribution). ...
26
votes
6answers
16k views

How to calculate Zipf's law coefficient from a set of top frequencies?

I have several query frequencies, and I need to estimate the coefficient of Zipf's law. These are the top frequencies: ...
7
votes
3answers
5k views

Is principal components analysis valid if the distribution(s) are Zipf like? What would be similar to PCA but suited to non gaussian data?

I'm analyzing people based on their twitter stream. We are using a 'word bag' model of users, which basically amounts to counting how often each word appears in a persons twitter stream (and then ...