54
$\begingroup$

I know, this may sound like it is off-topic, but hear me out.

At Stack Overflow and here we get votes on posts, this is all stored in a tabular form.

E.g.:

post id     voter id    vote type     datetime
-------     --------    ---------     --------
10          1           2             2000-1-1 10:00:01 
11          3           3             2000-1-1 10:00:01 
10          5           2             2000-1-1 10:00:01 

... and so on. Vote type 2 is an upvote, vote type 3 is a downvote. You can query an anonymized version of this data at http://data.stackexchange.com

There is a perception that if a post reaches the score of -1 or lower it is more likely to be upvoted. This may be simply confirmation bias or it may be rooted in fact.

How would we analyze this data to confirm or deny this hypothesis? How would we measure the effect of this bias?

$\endgroup$
7
  • 2
    $\begingroup$ can we get an example of the query? Not everybody is well versed in writing SQL statements. Having sample data might encourage people to try to play with it. +1 for the question. $\endgroup$
    – mpiktas
    Commented Jun 1, 2011 at 6:45
  • $\begingroup$ @Jeff votes are anonymized you can only get partial info from the data dump, it does include all transitions though here is a quick sample data.stackexchange.com/stackoverflow/q/101738 full anonymized data is available in the public data dump $\endgroup$ Commented Jun 1, 2011 at 8:30
  • $\begingroup$ Why just upvotes? How the probability of up- or down-voting splits around each particular value would be interesting surely? $\endgroup$ Commented Jun 1, 2011 at 8:51
  • $\begingroup$ @Bob, sure agree they would $\endgroup$ Commented Jun 1, 2011 at 9:35
  • 2
    $\begingroup$ I've seen other kinds of sites obfuscate votes (i.e. add noise before displaying them) and sometimes even completely hide up- and down-votes for a short period, in order to avoid various forms of bandwagonning, pity votes and other 'social' elements of voting. $\endgroup$
    – Glen_b
    Commented Jun 14, 2013 at 1:55

3 Answers 3

37
$\begingroup$

You could use a multistate model or Markov chain (the msm package in R is one way to fit these). You could then look to see if the transition probability from -1 to 0 is greater than from 0 to 1, 1 to 2, etc. You can also look at the average time at -1 compared to the others to see if it is shorter.

$\endgroup$
3
  • 4
    $\begingroup$ +1 great reference. There is an article in Journal of Statistical Software about msm package. The model seems ideally fitted for this kind of task. $\endgroup$
    – mpiktas
    Commented Jun 1, 2011 at 6:51
  • 4
    $\begingroup$ The Markov chain model idea looks like a good one, but the average time at -1 won't give the whole story. It's possible (and plausible - think bad questions) that one is more likely to get downvoted at -1 than elsewhere too. $\endgroup$ Commented Jun 1, 2011 at 8:51
  • $\begingroup$ I guess what one may want to do first is cluster the vote-trajectories - those that get (almost) only up/downvoted (very popular/very bad questions), and those that are more contentious. Then you can do Markov chains on the three classes. $\endgroup$
    – Jonas
    Commented Jun 1, 2011 at 13:52
14
$\begingroup$

Conduct an experiment. Randomly downvote half of the new posts at a particular time every day.

$\endgroup$
4
  • 6
    $\begingroup$ Cool, we should observe a significant increase in "critic" badges and probably a decrease in motivation for new users :-) Better to start with high-rep users, in this case (at risk of biasing the experiment!) $\endgroup$
    – chl
    Commented Jun 1, 2011 at 20:09
  • 16
    $\begingroup$ Actually we could do better than this ... using AB testing we could pick to display half of the -1 voted question on the site as 0 and half as -1 ... and see if either of the groups is more likely to be upvoted! Ingenious. $\endgroup$ Commented Jun 2, 2011 at 7:41
  • 5
    $\begingroup$ The experiment idea controls the quality of the posts, but (1) those being downgraded should agree in advance to participate in the experiment, and (2) after a brief time, the downgrades should be removed. $\endgroup$
    – zbicyclist
    Commented Jun 4, 2011 at 20:00
  • 3
    $\begingroup$ +1 (and +1 to all comments here, too): a controlled reversible experiment, communicated in advance to all users who might be affected and conducted with their approval, is one of the strongest ways to obtain this information. $\endgroup$
    – whuber
    Commented Jun 21, 2012 at 20:21
13
$\begingroup$

Summary of my answer. I like the Markov chain modeling but it misses the "temporal" aspect. On the other end, focusing on the temporal aspect (e.g. average time at $-1$) misses the "transition" aspect. I would go into the following general modelling (which with suitable assumption can lead to [markov process][1]). Also there is a lot of "censored" statistic behind this problem (which is certainly a classical problem of Software reliability ? ). The last equation of my answer gives the maximum likelihood estimator of voting intensity (up with "+" and dow with "-") for a given state of vote. As we can see from the equation, it is an intermediate from the case when you only estimate transition probability and the case when you only measure time spent at a given state. Hope this help.

General Modelling (to restate the question and assumptions). Let $(VD_i)_{i\geq 1}$ and $(S_{i})_{i\geq 1}$ be random variables modelling respectively the voting dates and the associated vote sign (+1 for upvote, -1 for downvote). The voting process is simply

$$Y_{t}=Y^+_t-Y^-_t$$ where

$$Y^+_t=\sum_{i=0}^{\infty}1_{VD_i\leq t,S_i=1} \;\text{ and } \;Y^-_t=\sum_{i=0}^{\infty}1_{VD_i\leq t,S_i=-1}$$

The important quantity here is the intentity of $\epsilon$-jump $$\lambda^{\epsilon}_t=\lim_{dt\rightarrow 0} \frac{1}{dt} P(Y^{\epsilon}_{t+dt}-Y^{\epsilon}_t=1|\mathcal{F}_t) $$ where $\epsilon$ can be $-$ or $+$ and $\mathcal{F}_t$ is a good filtration, in the genera case, without other knowledge it would be: $$\mathcal{F}_t=\sigma \left (Y^+_t,Y^-_t,VD_1,\dots,VD_{Y^+_t+Y^-_t},S_{1},\dots,S_{Y^+_t+Y^-_t} \right )$$.

but along the lines of your question, I think you implicitly assume that $$ P \left ( Y^{\epsilon}_{t+dt}-Y^{\epsilon}_t=1 | \mathcal{F}_t \right )= P \left (Y^{\epsilon}_{t+dt}-Y^{\epsilon}_t=1| Y_t \right ) $$ This means that for $\epsilon=+,-$ there exists a deterministic sequence $(\mu^{\epsilon}_i)_{i\in \mathbb{Z}}$ such that $\lambda^{\epsilon}_t=\mu^{\epsilon}_{Y_t}$.

Within this formalism, you question can be restated as: "it likely that $ \mu^{+}_{-1} -\mu^{+}_{0}>0$ " (or at least is the difference larger than a given threshold).

Under this assumption, it is easy to show that $Y_t$ is an [homogeneous markov process][3] on $\mathbb{Z}$ with generator $Q$ given by

$$\forall i,j \in \mathbb{Z}\;\;\; Q_{i,i+1}=\mu^{+}_{i}\;\; Q_{i,i-1}=\mu^{-}_{i}\;\; Q_{ii}=1-(\mu^{+}_{i}+\mu^{-}_{i}) \;\; Q_{ij}=0 \text{ if } |i-j|>1$$

Answering the question (through proposing a maximum likelihood estimatior for the statistical problem) From this reformulation, solving the problem is done by estimating $(\mu^{+}_i)$ and building a test uppon its values. Let us fix and forget the $i$ index without loss of generality. Estimation of $\mu^+$ (and $\mu^-$) can be done uppon the observation of

$(T^{1},\eta^1),\dots,(T^{p},\eta^p)$ where $T^j$ are the lengths of the $j^{th}$ of the $p$ periods spent in state $i$ (i.e. successive times with $Y_t=i$) and $\eta^j$ is $+1$ if the question was upvoted, $-1$ if it was downvoted and $0$ if it was the last state of observation.

If you forget the case with the last state of observation, the mentionned couples are iid from a distribution that depends on $\mu_i^+$ and $\mu_i^-$: it is distributed as $(\min(Exp(\mu_i^+),Exp(\mu_i^-)),\eta)$ (where Exp is a random var from an exponential distribution and $\eta$ is + or -1 depending on who realizes the max). Then, you can use the following simple lemma (the proof is straightforward):

Lemma If $X_+\leadsto Exp(\mu_+)$ and $X_{-} \leadsto Exp(\mu_{-})$ then, $T=\min(X_+,X_-)\leadsto Exp(\mu_++\mu_-)$ and $P(X_+1<X_-)=\frac{\mu_+}{\mu_++\mu_-}$.

This implies that the density $f(t,\epsilon)$ of $(T,\eta)$ is given by: $$ f(t,\epsilon)=g_{\mu_++\mu_-}\left ( \frac{1(\epsilon=+1)*\mu_++1(\epsilon=-1)*\mu_-}{\mu_++\mu_-}\right )$$ where $g_a$ for $a>0$ is the density function of an exponential random variable with parameter $a$. From this expression, it is easy to derive the maximum likelihood estimator of $\mu_+$ and $\mu_-$:

$$(\hat{\mu}_+,\hat{\mu}_-)=argmin \ln (\mu_-+\mu_+)\left ( (\mu_-+\mu_+)\sum_{i=1}^p T^i+p\right )- p_-\ln\left (\mu_-\right ) -p_+ \ln \left (\mu_+\right )$$ where $p_-=|{i:\delta_i=-1}|$ and $p_+=|{i:\delta_i=+1}|$.

Comments for more advanced approaches

If you want to take into acount cases when $i$ is the last observed state (certainly smarter because when you go through $-1$, it is often your last score...) you have to modify a little bit the reasonning. The corresponding censoring is relatively classical...

Possible other approache may include the possibility of

  • Having an intensity that decreases in with time
  • Having an intensity that decreases with the time spent since the last vote (I prefer this one. In this case there are classical way of modelling how the density decreases...
  • You may want to assume that $\mu_i^+$ is a smooth function of $i$
  • .... you can propose other ideas !
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.