I have a dataset which is statistics from a web discussion forum. I'm looking at the distribution of the number of replies a topic is expected to have. In particular, I've created a dataset which has a list of topic reply counts, and then the count of topics which have that number of replies.
"num_replies","count"
0,627568
1,156371
2,151670
3,79094
4,59473
5,39895
6,30947
7,23329
8,18726
If I plot the dataset on a log-log plot, I get what is basically a straight line:
(This is a Zipfian distribution). Wikipedia tells me that straight lines on log-log plots imply a function that can be modelled by a monomial of the form $y = ax^k$. And in fact I've eyeballed such a function:
lines(data$num_replies, 480000 * data$num_replies ^ -1.62, col="green")
My eyeballs obviously aren't as accurate as R. So how can I get R to fit the parameters of this model for me more accurately? I tried polynomial regression, but I don't think that R tries to fit the exponent as a parameter - what is the proper name for the model I want?
Edit: Thanks for the answers everyone. As suggested, I've now fit a linear model against the logs of the input data, using this recipe:
data <- read.csv(file="result.txt")
# Avoid taking the log of zero:
data$num_replies = data$num_replies + 1
plot(data$num_replies, data$count, log="xy", cex=0.8)
# Fit just the first 100 points in the series:
model <- lm(log(data$count[1:100]) ~ log(data$num_replies[1:100]))
points(data$num_replies, round(exp(coef(model)[1] + coef(model)[2] * log(data$num_replies))),
col="red")
The result is this, showing the model in red:
That looks like a good approximation for my purposes.
If I then use this Zipfian model (alpha = 1.703164) along with a random number generator to generate the same total number of topics (1400930) as the original measured dataset contained (using this C code I found on the web), the result looks like:
Measured points are in black, randomly generated ones according to the model are in red.
I think this shows that the simple variance created by randomly generating these 1400930 points is a good explanation for the shape of the original graph.
If you're interested in playing with the raw data yourself, I have posted it here.