How can I improve my analysis of the effects of reputation on voting?

Question

Recently I had done some analysis of the effects of reputation on upvotes (see the blog-post), and subsequently I had a few questions about possibly more enlightening (or more appropriate) analysis and graphics.

So a few questions (and feel free to respond to anyone in particular and ignore the others):

1. In its current in incarnation, I did not mean center the post number. I think what this does is give the false appearance of a negative correlation in the scatterplot, as there are more posts towards the lower end of the post count (you see this doesn't happen in the Jon Skeet panel, only in the mortal users panel). Is it innapropriate to not mean-center the post number (since I mean centered the score per user average score)?

2. It should be obvious from the graphs that score is highly right skewed (and mean centering did not change that any). When fitting a regression line, I fit both linear models and a model using the Huber-White sandwhich errors (via rlm in the MASS R package) and it did not make any difference in the slope estimates. Should I have considered a transformation to the data instead of robust regression? Note that any transformation would have to take into account the possibility of 0 and negative scores. Or should I have used some other type of model for count data instead of OLS?

3. I believe the last two graphics, in general, could be improved (and is related to improved modelling strategies as well). In my (jaded) opinion, I would suspect if reputation effects are real they would be realized quite early on in a posters history (I suppose if true, these may be reconsidered "you gave some excellent answers so now I will upvote all of your posts" instead of "reputation by total score" effects). How can I create a graphic to demonstrate whether this is true, while taking into account for the over-plotting? I thought maybe a good way to demonstrate this would be to fit a model of the form;

$$Y = \beta_0 + \beta_1(X_1) + \alpha_1(Z_1) + \alpha_2(Z_2) \cdots \alpha_k(Z_k) + \gamma_1(Z_1*X_1) \cdots \gamma_k(Z_k*X_1) + \epsilon $$

where $Y$ is the score - (mean score per user) (the same as is in the current scatterplots), $X_1$ is the post number, and the $Z_1 \cdots Z_k$ are dummy variables representing some arbitrary range of post numbers (for example $Z_1$ equals 1 if the post number is 1 through 25, $Z_2$ equals 1 if the post number is 26 through 50 etc.). $\beta_0$ and $\epsilon$ are the grand intercept and error term respectively. Then I would just examine the estimated $\gamma$ slopes to determine if reputation effects appeared early on in a posters history (or graphically display them). Is this a reasonable (and appropriate) approach?

It seems popular to fit some type of non-parametric smoothing line to scatterplots like these (such as loess or splines), but my experimentation with splines did not reveal anything enlightening (any evidence of postive effects early on in poster history was slight and tempermental to the number of splines I included). Since I have a hypothesis that the effects happen early on, is my modelling approach above more reasonable than splines?

Also note although I've pretty much dredged all of this data, there are still plenty of other communities out there to examine (and some like superuser and serverfault have similarly large samples to draw from), so it is plenty reasonable to suggest in future analysis that I use a hold-out sample to examine any relationship.

Answer 1

This is a brave try, but with these data alone, it will be difficult or impossible to answer your research question concerning the "effect of reputation on upvotes." The problem lies in separating the effects of other phenomena, which I list along with brief indications of how they might be addressed.

• Learning effects. As reputation goes up, experience goes up; as experience goes up, we would expect a person to post better questions and answers; as their quality improves, we expect more votes per post. Conceivably, one way to handle this in an analysis would be to identify people who are active on more than one SE site. On any given site their reputation would increase more slowly than the amount of their experience, thus providing a handle for teasing apart the reputation and learning effects.

• Temporal changes in context. These are myriad, but the obvious ones would include

  • Changes in numbers of voters over time, including an overall upward trend, seasonal trends (often associated with academic cycles), and outliers (arising from external publicity such as links to specific threads). Any analysis would have to factor this in when evaluating trends in reputation for any individual.

  • Changes in a community's mores over time. Communities, and how they interact, evolve and develop. Over time they may tend to vote more or less often. Any analysis would have to evaluate this effect and factor it in.

  • Time itself. As time goes by, earlier posts remain available for searching and continue to garner votes. Thus, caeteris paribus, older posts ought to produce more votes than newer ones. (This is a strong effect: some people consistently high on the monthly reputation leagues have not visited this site all year!) This would mask or even invert any actual positive reputation effect. Any analysis needs to factor in the length of time each post has been present on the site.

• Subject popularity. Some tags (e.g., r) are far more popular than others. Thus, changes in the kinds of questions a person answers can be confounded with temporal changes, such as a reputation effect. Therefore, any analysis needs to factor in the nature of the questions being answered.

• Views [added as an edit]. Questions are viewed by different numbers of people for various reasons (filters, links, etc.). It's possible the number of votes received by answers are related to the number of views, although one would expect a declining proportion as the number of views increases. (It's a matter of how many people who are truly interested in the question actually view it, not the raw number. My own--anecdotal--experience is that roughly half the upvotes I receive on many questions come within the first 5-15 views, although eventually the questions are viewed hundreds of times.) Therefore, any analysis needs to factor in the number of views, but probably not in a linear way.

• Measurement difficulties. "Reputation" is the sum of votes received for different activities: initial reputation, answers, questions, approving questions, editing tag wikis, downvoting, and getting downvoted (in descending order of value). Because these components assess different things, and not all are under the control of the community voters, they should be separated for analysis. A "reputation effect" presumably is associated with upvotes on answers and, perhaps, on questions, but should not affect other sources of reputation. The starting reputation definitely should be subtracted (but perhaps could be used as a proxy for some initial amount of experience).

• Hidden factors. There can be many other confounding factors that are impossible to measure. For example, there are various forms of "burnout" in participation in forums. What do people do after an initial few weeks, months, or years of enthusiasm? Some possibilities include focusing on the rare, unusual, or difficult questions; providing answers only to unanswered questions; providing fewer answers but of higher quality; etc. Some of these could mask a reputation effect, whereas others could mistakenly be confused with one. A proxy for such factors might be changes in rates of participation by an individual: they could signal changes in the nature of that person's posts.

• Subcommunity phenomena. A hard look at the statistics, even on very active SE pages, shows that a relatively small number of people do most of the answering and voting. A clique as small as two or three people can have a profound influence on the growth of reputation. A two-person clique will be detected by the site's built-in monitors (and one such group exists on this site), but larger cliques probably won't be. (I'm not talking about formal collusion: people can be members of such cliques without even being aware of it.) How would we separate an apparent reputation effect from activities of these invisible, undetected, informal cliques? Detailed voting data could be used diagnostically, but I don't believe we have access to these data.

• Limited data. To detect a reputation effect, you will likely need to focus on individuals with dozens to hundreds of posts (at least). That drops the current population to less than 50 individuals. With all the possibility of variation and confounding, that is far too small to tease out significant effects unless they are very strong indeed. The cure is to augment the dataset with records from other SE sites.

Given all these complications, it should be clear that the exploratory graphics in the blog article have little chance of revealing anything unless it is glaringly obvious. Nothing leaps out at us: as expected, the data are messy and complicated. It's premature to recommend improvements to the plots or to the analysis that has been presented: incremental changes and additional analysis won't help until these fundamental issues have been addressed.

Answer 2

Econometricians have looked at similar issues within the framework of Granger causality. If you have two series, $Y_t$ and $Z_t$, you can run vector autoregressive models, which in the simplest form with a single lag look like $Y_t = a_0 + a_1 Y_{t-1} + a_2 Z_{t-1} + \epsilon_t$, $Z_t = b_0 + b_1 Y_{t-1} + b_2 Z_{t-1} + \delta_t$. If you see that say $a_2$ is significant, then you can claim that $Z$ (Granger-)causes $Y$: adding information about $Z$ improves the precision of your model for $Y$. Here, your time $t$ would be the post number, and the variables are obviously reputation and the score. Both are non-stationary, so a more serious fiddling with the data, like taking the increments $\Delta Y_t = Y_t - Y_{t-1}$ in place of $Y_t$ in the above equations will be called for. (Note that you may lose the normal and normal-based $F$ or $\chi^2$ distributions with non-stationary data, and the rate of convergence with trend variables, if you include them into analysis, may be $T^{-1}$ or even faster, rather than $T^{-1/2}$ that most of us are used to from the Central Limit Theorem. You need to be super-careful with these.) So I guess if $Y_t$ is the answer score, and $Z_t$ is reputation, then clearly $a_0$ is the average score, $a_1$ is how the person learns to write better answers, and $a_2$ is how their reputation precedes their word (provided the model assumptions are satisfied, etc.)

On point 1: if you were doing fixed effects by hand, you should've centered both the response variable and the explanatory variables. The panel data regression package would've done this for you, but the official econometric way of looking at things is to subtract the "between" regression from the "pooled" regression (see Wooldridge's black book; I have not checked the second edition, but I generally view the first edition as the best textbook-type description of econometric panel data).

On your point 2: of course Eicker/White standard errors won't affect your point estimates; if they did, that would indicate an incorrect implementation! In the context of time-series, an even more appropriate estimator is due to Newey and West (1987). Trying transformations might help. I am personally a big fan of the Box-Cox transformation, but in the context of the analysis that you are undertaking, it is difficult to do it cleanly. First, you would need a shift parameter on top of the shape parameter, and the shift parameters are notoriously difficult to identify in models like this. Second, you would probably need different shift/shape parameters for different people, and/or different posts, and/or... (all the hell breaking loose). Count data is an option, too, but in the context of mean modeling, a Poisson regression is just as good as the log transformation, yet it imposes an unwieldy assumption of variance = mean.

P.S. You could probably tag this with "longitudinal-data" and "time-series".

Answer 3

Several other changes to plots:

1. Quantile bands for the answer score versus previous reputation. (Plots 1 & 3)
2. Density plots for Skeet versus others, stratified by post # (Plot 3)
3. Consider stratifying by # of competing posts
4. Stratify by time (one may continue to gain points long after the question has been asked)

Modeling this will be harder. You might consider Poisson regression. Frankly, though, developing good plots is a much better method of developing insights and skills. Begin modeling after you have a better understanding of the data.

Answer 4

Whoa there. (And I mean that in a good way ;-)) Before going further with models, you need to address what's going on with the data.

I don't see an explanation for the very peculiar curve in the middle of this plot: http://stats.blogoverflow.com/files/2011/07/Rep_Correlated_With_Upvotes.png

Seeing such a curve makes me think that there's something very weird about those points - that they're not independent from each other and instead reflect some sequence of observations of the same source.

(Minor note: titling that plot "Correlation..." is misleading.)