The tag has no usage guidance.

learn more… | top users | synonyms

0
votes
0answers
18 views

Frequency of the most frequent word using Zipf's law

I modeled my corpus using Zipf's law, so that: $$f_r=c/r$$ Where $f_r$ is the frequency of a word at rank $r$. I calculated $c=90000$. How can I calculate the frequency for the most frequent word? ...
1
vote
0answers
28 views

Understanding Zipf's law calculation for probability of a word

So, I am analyzing a a corpus of ~1,000,000 words and I am trying to understand how I can use Zipf's law. I found on some textbook slides the following formulas: $$f_r = c/r$$ $$p_r = k/r $$ where ...
0
votes
0answers
35 views

Best textbooks on Non-Homogeneous Stochastic Processes?

just wanted to know which are in your opinion some of the best available books on theory and applications of NH Poisson Stochastic Processes, and Non-Poisson processes out there. I've studied Parzen ...
0
votes
0answers
50 views

Fitting multiple power laws, Zipf's law in the real-world

As a preface, the following questions are related: How to calculate Zipf's law coefficient from a set of top frequencies? How to estimate parameters for Zipf truncated distribution from a data ...
2
votes
0answers
20 views

Exponent value in long tail phenomenon of online store

I've read (pdf) that there is a long-tail phenomenon (or generalized Zipf law) in popularity of retail items: If I sort items for sale by their popularity, I will see a long tail of not very popular ...
1
vote
0answers
17 views

Estimating distribution of traffic across all Web pages

The web is estimated to have approx 5 billion web pages ( http://www.worldwidewebsize.com/ ). How would you go about estimating a CDF to know how many pages are required that make up, say, 90% of ...
1
vote
1answer
253 views

Connection between power law and Zipf's law

I am trying to better understand the connection between the power law distribution and Zipf's distribution (law). There is a neat explanation in [1]. The article suggests that as we can derivate the ...
1
vote
0answers
153 views

Using Zipf's law to select top K ngrams?

Lets say we extract all the ngrams (unigrams, bigrams, trigrams) along with their frequency from a text. Now if we want to select say some Top K ngrams, is there a way to estimate a good value of K ...
4
votes
0answers
63 views

Zipf's vs Self-similar: are they really the same

Recently I ran into a test using both zipf's and self-similar generated datasets. I followed the description from Jim Gray's paper on generating such datasets (Quickly Generating Billon-Record ...
2
votes
0answers
194 views

Sales prediction of Amazon Sales Rank

I understand that Zipf's Rank-size distribution would allow me to estimate the sales of a book given the amazon sales rank. How can I fit the distribution without knowing book sales leading to a rank. ...
0
votes
1answer
490 views

How to calculate zipf exponent of page degree distribution?

I want to calculate the exponent of the function which shows zipf's law in data. I'm working in the context of graph node degree analysis and want to comprehend results shown by Border et al. on a new ...
0
votes
1answer
281 views

Discretize frequency of words that follow zipfian distribution

How could I discretize the frequency of words found in a corpus that follow a zipfian distribution? Are there standard methods? It should create bins of exponential-increasing size. My goal is to ...
1
vote
0answers
363 views

Mean and median in Zipf's distribution

I have a collection of $10^5$ essays, each of which on average contains $10^3$ distinct words. There are $10^6$ distinct words in the entire collection. If I index every word what is the mean and ...
8
votes
3answers
2k views

How to estimate parameters for Zipf truncated distribution from a data sample?

I have a problem with the estimation parameter for Zipf. My situation is the following: I have a sample set (measured from an experiment that generates calls that should follow a Zipf distribution). ...
15
votes
2answers
6k views

How to calculate Zipf's law coefficient from a set of top frequencies?

I have several query frequencies, and I need to estimate the coefficient of Zipf's law. These are the top frequencies: ...
6
votes
1answer
2k views

Is principal components analysis valid if the distribution(s) are Zipf like? What would be similar to PCA but suited to non gaussian data?

I'm analyzing people based on their twitter stream. We are using a 'word bag' model of users, which basically amounts to counting how often each word appears in a persons twitter stream (and then ...