15
$\begingroup$

Recently I had done some analysis of the effects of reputation on upvotes (see the blog-post), and subsequently I had a few questions about possibly more enlightening (or more appropriate) analysis and graphics.

So a few questions (and feel free to respond to anyone in particular and ignore the others):

  1. In its current in incarnation, I did not mean center the post number. I think what this does is give the false appearance of a negative correlation in the scatterplot, as there are more posts towards the lower end of the post count (you see this doesn't happen in the Jon Skeet panel, only in the mortal users panel). Is it innapropriate to not mean-center the post number (since I mean centered the score per user average score)?

  2. It should be obvious from the graphs that score is highly right skewed (and mean centering did not change that any). When fitting a regression line, I fit both linear models and a model using the Huber-White sandwhich errors (via rlm in the MASS R package) and it did not make any difference in the slope estimates. Should I have considered a transformation to the data instead of robust regression? Note that any transformation would have to take into account the possibility of 0 and negative scores. Or should I have used some other type of model for count data instead of OLS?

  3. I believe the last two graphics, in general, could be improved (and is related to improved modelling strategies as well). In my (jaded) opinion, I would suspect if reputation effects are real they would be realized quite early on in a posters history (I suppose if true, these may be reconsidered "you gave some excellent answers so now I will upvote all of your posts" instead of "reputation by total score" effects). How can I create a graphic to demonstrate whether this is true, while taking into account for the over-plotting? I thought maybe a good way to demonstrate this would be to fit a model of the form;

$$Y = \beta_0 + \beta_1(X_1) + \alpha_1(Z_1) + \alpha_2(Z_2) \cdots \alpha_k(Z_k) + \gamma_1(Z_1*X_1) \cdots \gamma_k(Z_k*X_1) + \epsilon $$

where $Y$ is the score - (mean score per user) (the same as is in the current scatterplots), $X_1$ is the post number, and the $Z_1 \cdots Z_k$ are dummy variables representing some arbitrary range of post numbers (for example $Z_1$ equals 1 if the post number is 1 through 25, $Z_2$ equals 1 if the post number is 26 through 50 etc.). $\beta_0$ and $\epsilon$ are the grand intercept and error term respectively. Then I would just examine the estimated $\gamma$ slopes to determine if reputation effects appeared early on in a posters history (or graphically display them). Is this a reasonable (and appropriate) approach?

It seems popular to fit some type of non-parametric smoothing line to scatterplots like these (such as loess or splines), but my experimentation with splines did not reveal anything enlightening (any evidence of postive effects early on in poster history was slight and tempermental to the number of splines I included). Since I have a hypothesis that the effects happen early on, is my modelling approach above more reasonable than splines?

Also note although I've pretty much dredged all of this data, there are still plenty of other communities out there to examine (and some like superuser and serverfault have similarly large samples to draw from), so it is plenty reasonable to suggest in future analysis that I use a hold-out sample to examine any relationship.

$\endgroup$
  • $\begingroup$ I've currently made some notes on my first question, and they can be found here. I'm not sure at the moment whether I should just post this as an answer to my own question or open a seperate question (as this is largely focused on data visualization). But feel free to leave me a comment about the google document either here or in the chat room. $\endgroup$ – Andy W Aug 8 '11 at 15:21
14
$\begingroup$

This is a brave try, but with these data alone, it will be difficult or impossible to answer your research question concerning the "effect of reputation on upvotes." The problem lies in separating the effects of other phenomena, which I list along with brief indications of how they might be addressed.

  • Learning effects. As reputation goes up, experience goes up; as experience goes up, we would expect a person to post better questions and answers; as their quality improves, we expect more votes per post. Conceivably, one way to handle this in an analysis would be to identify people who are active on more than one SE site. On any given site their reputation would increase more slowly than the amount of their experience, thus providing a handle for teasing apart the reputation and learning effects.

  • Temporal changes in context. These are myriad, but the obvious ones would include

    • Changes in numbers of voters over time, including an overall upward trend, seasonal trends (often associated with academic cycles), and outliers (arising from external publicity such as links to specific threads). Any analysis would have to factor this in when evaluating trends in reputation for any individual.

    • Changes in a community's mores over time. Communities, and how they interact, evolve and develop. Over time they may tend to vote more or less often. Any analysis would have to evaluate this effect and factor it in.

    • Time itself. As time goes by, earlier posts remain available for searching and continue to garner votes. Thus, caeteris paribus, older posts ought to produce more votes than newer ones. (This is a strong effect: some people consistently high on the monthly reputation leagues have not visited this site all year!) This would mask or even invert any actual positive reputation effect. Any analysis needs to factor in the length of time each post has been present on the site.

  • Subject popularity. Some tags (e.g., ) are far more popular than others. Thus, changes in the kinds of questions a person answers can be confounded with temporal changes, such as a reputation effect. Therefore, any analysis needs to factor in the nature of the questions being answered.

  • Views [added as an edit]. Questions are viewed by different numbers of people for various reasons (filters, links, etc.). It's possible the number of votes received by answers are related to the number of views, although one would expect a declining proportion as the number of views increases. (It's a matter of how many people who are truly interested in the question actually view it, not the raw number. My own--anecdotal--experience is that roughly half the upvotes I receive on many questions come within the first 5-15 views, although eventually the questions are viewed hundreds of times.) Therefore, any analysis needs to factor in the number of views, but probably not in a linear way.

  • Measurement difficulties. "Reputation" is the sum of votes received for different activities: initial reputation, answers, questions, approving questions, editing tag wikis, downvoting, and getting downvoted (in descending order of value). Because these components assess different things, and not all are under the control of the community voters, they should be separated for analysis. A "reputation effect" presumably is associated with upvotes on answers and, perhaps, on questions, but should not affect other sources of reputation. The starting reputation definitely should be subtracted (but perhaps could be used as a proxy for some initial amount of experience).

  • Hidden factors. There can be many other confounding factors that are impossible to measure. For example, there are various forms of "burnout" in participation in forums. What do people do after an initial few weeks, months, or years of enthusiasm? Some possibilities include focusing on the rare, unusual, or difficult questions; providing answers only to unanswered questions; providing fewer answers but of higher quality; etc. Some of these could mask a reputation effect, whereas others could mistakenly be confused with one. A proxy for such factors might be changes in rates of participation by an individual: they could signal changes in the nature of that person's posts.

  • Subcommunity phenomena. A hard look at the statistics, even on very active SE pages, shows that a relatively small number of people do most of the answering and voting. A clique as small as two or three people can have a profound influence on the growth of reputation. A two-person clique will be detected by the site's built-in monitors (and one such group exists on this site), but larger cliques probably won't be. (I'm not talking about formal collusion: people can be members of such cliques without even being aware of it.) How would we separate an apparent reputation effect from activities of these invisible, undetected, informal cliques? Detailed voting data could be used diagnostically, but I don't believe we have access to these data.

  • Limited data. To detect a reputation effect, you will likely need to focus on individuals with dozens to hundreds of posts (at least). That drops the current population to less than 50 individuals. With all the possibility of variation and confounding, that is far too small to tease out significant effects unless they are very strong indeed. The cure is to augment the dataset with records from other SE sites.

Given all these complications, it should be clear that the exploratory graphics in the blog article have little chance of revealing anything unless it is glaringly obvious. Nothing leaps out at us: as expected, the data are messy and complicated. It's premature to recommend improvements to the plots or to the analysis that has been presented: incremental changes and additional analysis won't help until these fundamental issues have been addressed.

$\endgroup$
  • $\begingroup$ Thank you for the response. Given the breadth of the critique, I won't be able to appropriately address all of the suggestions in comments (I will have to think of another venue, maybe just post another google document). But I will say now I don't think it is impossible to answer (to the extent that anyone can answer anything with observational data such as this). At a minimum, given the limitations of potential confounds, one can see if reputation effects are consistent with the evidence available. $\endgroup$ – Andy W Aug 16 '11 at 15:28
  • $\begingroup$ @Andy I think the confounding is substantial and pervasive, so that even if it looks like a reputation effect is there, it could be an artifact: you won't be able to draw any valid conclusion unless you have addressed these problems. Of course I could be wrong, but the burden of proof is on you. $\endgroup$ – whuber Aug 16 '11 at 15:37
  • $\begingroup$ the "if it looks like a reputation effect is there" is the key statement (as I see it). Most of the confounds you presented would either by ambiguously related to a posters reputation/post number/history or would be theoretically expected to increase the posters score on answers later on in their history. If I find no evidence of reputation effects, many of the potential confounds can't be used to explain its absence. $\endgroup$ – Andy W Aug 16 '11 at 15:41
  • $\begingroup$ @Andy But at least one can, and that's enough. These include hidden factors, subject popularity, and temporal changes in context. If you don't explicitly handle all of these in the analysis, your conclusions will be suspect. A glance at the records shows that subject popularity and temporal changes are huge; their potential influences swamp what we might reasonably expect reputation effects to be by up to an order of magnitude. $\endgroup$ – whuber Aug 16 '11 at 15:47
  • 2
    $\begingroup$ @cardinal , even without a formal definition, it would be possible for a small number of people to have an appreciable impact on voting patterns (which is what I assume whuber is referring to in this context). Jon Skeet's average post was only 5 something upvotes. If all of a sudden one person decides to upvote all of his answers, that could have a pretty substantial impact given the low average score to begin with. $\endgroup$ – Andy W Aug 17 '11 at 16:44
5
$\begingroup$

Econometricians have looked at similar issues within the framework of Granger causality. If you have two series, $Y_t$ and $Z_t$, you can run vector autoregressive models, which in the simplest form with a single lag look like $Y_t = a_0 + a_1 Y_{t-1} + a_2 Z_{t-1} + \epsilon_t$, $Z_t = b_0 + b_1 Y_{t-1} + b_2 Z_{t-1} + \delta_t$. If you see that say $a_2$ is significant, then you can claim that $Z$ (Granger-)causes $Y$: adding information about $Z$ improves the precision of your model for $Y$. Here, your time $t$ would be the post number, and the variables are obviously reputation and the score. Both are non-stationary, so a more serious fiddling with the data, like taking the increments $\Delta Y_t = Y_t - Y_{t-1}$ in place of $Y_t$ in the above equations will be called for. (Note that you may lose the normal and normal-based $F$ or $\chi^2$ distributions with non-stationary data, and the rate of convergence with trend variables, if you include them into analysis, may be $T^{-1}$ or even faster, rather than $T^{-1/2}$ that most of us are used to from the Central Limit Theorem. You need to be super-careful with these.) So I guess if $Y_t$ is the answer score, and $Z_t$ is reputation, then clearly $a_0$ is the average score, $a_1$ is how the person learns to write better answers, and $a_2$ is how their reputation precedes their word (provided the model assumptions are satisfied, etc.)

On point 1: if you were doing fixed effects by hand, you should've centered both the response variable and the explanatory variables. The panel data regression package would've done this for you, but the official econometric way of looking at things is to subtract the "between" regression from the "pooled" regression (see Wooldridge's black book; I have not checked the second edition, but I generally view the first edition as the best textbook-type description of econometric panel data).

On your point 2: of course Eicker/White standard errors won't affect your point estimates; if they did, that would indicate an incorrect implementation! In the context of time-series, an even more appropriate estimator is due to Newey and West (1987). Trying transformations might help. I am personally a big fan of the Box-Cox transformation, but in the context of the analysis that you are undertaking, it is difficult to do it cleanly. First, you would need a shift parameter on top of the shape parameter, and the shift parameters are notoriously difficult to identify in models like this. Second, you would probably need different shift/shape parameters for different people, and/or different posts, and/or... (all the hell breaking loose). Count data is an option, too, but in the context of mean modeling, a Poisson regression is just as good as the log transformation, yet it imposes an unwieldy assumption of variance = mean.

P.S. You could probably tag this with "longitudinal-data" and "time-series".

$\endgroup$
  • $\begingroup$ thank you for the response, and a few comments/questions. I agree I should have at least explored a more explicit time series approach in this data (I did not even check to see if there was any evidence of autocorrelation in the residuals). There are a few more complications though in time series modelling of this data (what is t?, and score itself is dynamic and not fixed per post number), also there would be no need for a regression predicting Z_t, I know perfectly what Z_t is a function of! $\endgroup$ – Andy W Aug 11 '11 at 12:49
  • $\begingroup$ Also I highly doubt score is non-stationary, what makes you think it is? $\endgroup$ – Andy W Aug 11 '11 at 12:52
  • $\begingroup$ At the very least, it is probably heteroskedastic: some posts are interesting, get a lot of hits and a lot of upvotes, while others are small clarifications or RTFM-"Read this link" type of questions/answers. That of itself would technically make it non-stationary. Of course stationarity is a testable assumption, but with crazy data like these, you'd probably want to be on a very safe side of being overly conservative in the analysis methods (or, as I mentioned, to be aware that the results may be weird). $\endgroup$ – StasK Aug 11 '11 at 13:22
  • $\begingroup$ I'm a bit confused by the last comment. How do exogenous factors that affect the score of an answer make the series heteroskedastic (I assume you mean that the variance of score gets larger/smaller with post number?), and of what relevance is this to the question at hand? $\endgroup$ – Andy W Aug 11 '11 at 14:51
  • $\begingroup$ A time series is stationary if the marginal distributions at all time points are the same. So even you might have the same mean, a changing variance will make the series non-stationary. An example are (G)ARCH models for which a Nobel prize was given in the early 2000s. But in these data, I would expect some shifts in the mean, as well. If the audience of the website grows, then for a given quality of an answer, you would likely to see more votes on it, which will likely raise both the mean and the variance of the scores. $\endgroup$ – StasK Aug 11 '11 at 15:36
3
$\begingroup$

Several other changes to plots:

  1. Quantile bands for the answer score versus previous reputation. (Plots 1 & 3)
  2. Density plots for Skeet versus others, stratified by post # (Plot 3)
  3. Consider stratifying by # of competing posts
  4. Stratify by time (one may continue to gain points long after the question has been asked)

Modeling this will be harder. You might consider Poisson regression. Frankly, though, developing good plots is a much better method of developing insights and skills. Begin modeling after you have a better understanding of the data.

$\endgroup$
  • $\begingroup$ (+1) After letting the post sit for awhile, I realized that visualizing the density of the points appears to be a much better solution than trying to visualize the points themselves (although I'm not quite sure what you mean by "stratify by the post #"). I also think plotting the estimated quantiles sounds like a good idea, although for plot 1 & 2 it will likely just be in the massive cloud. Again I don't know what "stratify by time" in this context means either, see Brad Larson's comment on the blog post and my response in regards to this. $\endgroup$ – Andy W Aug 6 '11 at 3:50
  • $\begingroup$ Also I highly doubt competing posts has anything to do with the observed relationships. Do you think people whom have high reputation posted in threads with more competing answers earlier in their history? Your suggestions about including other covariates seem to be conflicting with the suggestion to avoid modelling and focus on plots. $\endgroup$ – Andy W Aug 6 '11 at 3:55
  • $\begingroup$ The idea behind the competing posts is exploratory in nature. Motivation to answer has nothing to do with it. Regarding modeling, it's not that I'm against modeling per se, but that you're not yet ready to do it until you have a better understanding of the data. If you don't understand the data, you won't understand the models. $\endgroup$ – Iterator Aug 6 '11 at 5:07
  • $\begingroup$ By stratify by post #, I am suggesting that you bin the posts. It can be on an interval scale, such as 0-100 posts, 101-200, etc. Or on a quantile scale: split the users by those in the bottom 10%ile of total posts, 20%ile, etc. Because Skeet has so many posts, it is best to compare him to his peer group, but it's hard to compare him to a peer group of those with precisely the same # of posts - binning the data may help. $\endgroup$ – Iterator Aug 6 '11 at 5:09
  • $\begingroup$ Btw, for the stratification, you can use coplot(). $\endgroup$ – Iterator Aug 6 '11 at 5:29
1
$\begingroup$

Whoa there. (And I mean that in a good way ;-)) Before going further with models, you need to address what's going on with the data.

I don't see an explanation for the very peculiar curve in the middle of this plot: http://stats.blogoverflow.com/files/2011/07/Rep_Correlated_With_Upvotes.png

Seeing such a curve makes me think that there's something very weird about those points - that they're not independent from each other and instead reflect some sequence of observations of the same source.

(Minor note: titling that plot "Correlation..." is misleading.)

$\endgroup$
  • 5
    $\begingroup$ That curve looks weird because of the weird choice of scales on the axes. It reflects replies that have contributed the majority of a user's reputation: the one-post wonders. It's exponential because the y axis is linear while the x axis is logarithmic. You really should ignore everything associated with log reputations less than $2$ because for many users that's where their reputations start and you should consider almost anything for log reputations less than $3$ to be just noise. As such, 99% of this graphic is devoted to displaying that noise: there's not much information there. $\endgroup$ – whuber Aug 4 '11 at 21:14
  • $\begingroup$ That curve can be explained by the nature of how reputation is related to upvotes, and is likely people whom have posted one answer and gained all reputation from that sole answer (I can go into more detail on why that is likely the case if needed). If I had plotted current reputation minus the reputation from the most current post this would have taken care of that for the most part (also those observations don't have anything to do with the subsequent analysis). Do you care to elaborate on the correlation being misleading? $\endgroup$ – Andy W Aug 4 '11 at 21:15
  • $\begingroup$ @whuber , I don't think I would say anything below 10^3 is just noise. Surely a theory of reputation effects should be applicable to when reputation is absent. I also welcome any suggested improvements to the plots (there is not much info in any of the plots!) $\endgroup$ – Andy W Aug 4 '11 at 22:00
  • $\begingroup$ Thanks. For the title, there is no calculation of the correlation. It's just a scatter plot of marginal score versus reputation. Except, as you and @whuber mention, it's not really the marginal score: it should be deltaRep (or Rep(t) - Rep(t-1)) versus Rep(t-1). $\endgroup$ – Iterator Aug 4 '11 at 22:02
  • 1
    $\begingroup$ @Iterator , correct for the last statement (10 points per upvote), but it still appears you may be confused what I am plotting with the other statement. The Y axis is not reputation, but the number of upvotes for the most recent post (this is not necessarily Rep(t) - Rep(t-1) as users can gain reputation from other places), the X axis is the current reputation (including reputation gained from that post). The X axis is what I suggested should be replaced (subtracting the upvotes gained from the answer in question I plotted on the Y axis). $\endgroup$ – Andy W Aug 4 '11 at 22:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.