Recently I had done some analysis of the effects of reputation on upvotes (see the blog-post), and subsequently I had a few questions about possibly more enlightening (or more appropriate) analysis and graphics.
So a few questions (and feel free to respond to anyone in particular and ignore the others):
In its current in incarnation, I did not mean center the post number. I think what this does is give the false appearance of a negative correlation in the scatterplot, as there are more posts towards the lower end of the post count (you see this doesn't happen in the Jon Skeet panel, only in the mortal users panel). Is it innapropriate to not mean-center the post number (since I mean centered the score per user average score)?
It should be obvious from the graphs that score is highly right skewed (and mean centering did not change that any). When fitting a regression line, I fit both linear models and a model using the Huber-White sandwhich errors (via
rlmin the MASS R package) and it did not make any difference in the slope estimates. Should I have considered a transformation to the data instead of robust regression? Note that any transformation would have to take into account the possibility of 0 and negative scores. Or should I have used some other type of model for count data instead of OLS?
I believe the last two graphics, in general, could be improved (and is related to improved modelling strategies as well). In my (jaded) opinion, I would suspect if reputation effects are real they would be realized quite early on in a posters history (I suppose if true, these may be reconsidered "you gave some excellent answers so now I will upvote all of your posts" instead of "reputation by total score" effects). How can I create a graphic to demonstrate whether this is true, while taking into account for the over-plotting? I thought maybe a good way to demonstrate this would be to fit a model of the form;
$$Y = \beta_0 + \beta_1(X_1) + \alpha_1(Z_1) + \alpha_2(Z_2) \cdots \alpha_k(Z_k) + \gamma_1(Z_1*X_1) \cdots \gamma_k(Z_k*X_1) + \epsilon $$
where $Y$ is the
score - (mean score per user) (the same as is in the current scatterplots), $X_1$ is the
post number, and the $Z_1 \cdots Z_k$ are dummy variables representing some arbitrary range of post numbers (for example $Z_1$ equals
1 if the post number is
1 through 25, $Z_2$ equals
1 if the post number is
26 through 50 etc.). $\beta_0$ and $\epsilon$ are the grand intercept and error term respectively. Then I would just examine the estimated $\gamma$ slopes to determine if reputation effects appeared early on in a posters history (or graphically display them). Is this a reasonable (and appropriate) approach?
It seems popular to fit some type of non-parametric smoothing line to scatterplots like these (such as loess or splines), but my experimentation with splines did not reveal anything enlightening (any evidence of postive effects early on in poster history was slight and tempermental to the number of splines I included). Since I have a hypothesis that the effects happen early on, is my modelling approach above more reasonable than splines?
Also note although I've pretty much dredged all of this data, there are still plenty of other communities out there to examine (and some like superuser and serverfault have similarly large samples to draw from), so it is plenty reasonable to suggest in future analysis that I use a hold-out sample to examine any relationship.