The tag has no usage guidance.

learn more… | top users | synonyms

2
votes
1answer
63 views

Fitting Zipf Mandelbrot and use Chi-square test in R

I have a dataset with hashtags and their frequencies (~370k frequencies), as for example (after a sort): ...
0
votes
1answer
18 views

How to asses the optimal bag of words vector size?

I have a corpus with 6040592 words and 309074 types (different words). Knowing this information it is possible to know the optimal size of bag of words vectors in order to represent phrases? I am ...
1
vote
0answers
18 views

Distinct items in a Zipfian distribution

I recently generated synthetic data sets with presumably Zipfian distribution and noticed that the frequency of the ranks doesn't change whether I generate 10^3 items or 10^6 items or a different ...
0
votes
1answer
28 views

Are Power Laws closed under monotonic transformation?

Let $X$ be a random variable that obeys power law. Let $m(\cdot)$ be a monotonic function. Does $Y=m(X)$ also obey power law? Can we then extend this to say that if $X$ follows Zipf's law, $m(X)$ also ...
3
votes
2answers
292 views

How to verify if data follows Zipf's law without looking at the graph

I want to check if a given text sample was written by real people or not so I think Zipf's law could help. If data follows Zipfian distribution, the most frequent word will occur approximately twice ...
2
votes
1answer
87 views

Understanding Zipf's law calculation for probability of a word

So, I am analyzing a a corpus of ~1,000,000 words and I am trying to understand how I can use Zipf's law. I found on some textbook slides the following formulas: $$f_r = c/r$$ $$p_r = k/r $$ where $...
2
votes
0answers
25 views

Exponent value in long tail phenomenon of online store

I've read (pdf) that there is a long-tail phenomenon (or generalized Zipf law) in popularity of retail items: If I sort items for sale by their popularity, I will see a long tail of not very popular ...
1
vote
0answers
27 views

Estimating distribution of traffic across all Web pages

The web is estimated to have approx 5 billion web pages ( http://www.worldwidewebsize.com/ ). How would you go about estimating a CDF to know how many pages are required that make up, say, 90% of ...
1
vote
1answer
681 views

Connection between power law and Zipf's law

I am trying to better understand the connection between the power law distribution and Zipf's distribution (law). There is a neat explanation in [1]. The article suggests that as we can derivate the ...
1
vote
0answers
221 views

Using Zipf's law to select top K ngrams?

Lets say we extract all the ngrams (unigrams, bigrams, trigrams) along with their frequency from a text. Now if we want to select say some Top K ngrams, is there a way to estimate a good value of K ...
4
votes
0answers
79 views

Zipf's vs Self-similar: are they really the same

Recently I ran into a test using both zipf's and self-similar generated datasets. I followed the description from Jim Gray's paper on generating such datasets (Quickly Generating Billon-Record ...
2
votes
0answers
242 views

Sales prediction of Amazon Sales Rank

I understand that Zipf's Rank-size distribution would allow me to estimate the sales of a book given the amazon sales rank. How can I fit the distribution without knowing book sales leading to a rank. ...
0
votes
1answer
564 views

How to calculate zipf exponent of page degree distribution?

I want to calculate the exponent of the function which shows zipf's law in data. I'm working in the context of graph node degree analysis and want to comprehend results shown by Border et al. on a new ...
0
votes
1answer
309 views

Discretize frequency of words that follow zipfian distribution

How could I discretize the frequency of words found in a corpus that follow a zipfian distribution? Are there standard methods? It should create bins of exponential-increasing size. My goal is to ...
1
vote
0answers
386 views

Mean and median in Zipf's distribution

I have a collection of $10^5$ essays, each of which on average contains $10^3$ distinct words. There are $10^6$ distinct words in the entire collection. If I index every word what is the mean and ...
9
votes
3answers
3k views

How to estimate parameters for Zipf truncated distribution from a data sample?

I have a problem with the estimation parameter for Zipf. My situation is the following: I have a sample set (measured from an experiment that generates calls that should follow a Zipf distribution). ...
20
votes
3answers
8k views

How to calculate Zipf's law coefficient from a set of top frequencies?

I have several query frequencies, and I need to estimate the coefficient of Zipf's law. These are the top frequencies: ...
6
votes
3answers
3k views

Is principal components analysis valid if the distribution(s) are Zipf like? What would be similar to PCA but suited to non gaussian data?

I'm analyzing people based on their twitter stream. We are using a 'word bag' model of users, which basically amounts to counting how often each word appears in a persons twitter stream (and then ...