12
$\begingroup$

There is an NPR podcast called Intelligence Squared. Each episode is a broadcasting of a live debate on some contentious statement such as "The 2nd amendment is no longer relevant" or "Affirmative action on college campuses does more harm than good". Four representatives debate-- two for the motion and two against.

To determine which side wins, the audience is polled both before and after the debate. The side that gained more in terms of absolute percentage is deemed the winner. For example:

          For    Against  Undecided
 Before   18%      42%       40%
 After    23%      49%       28%

 Winner: Against team -- The motion is rejected.

Intuitively, I think this measure of success is biased and I am wondering how one would poll the audience to determine the winner in a fair way.

Three issues I immediately see with the current method:

  • At the extremes, if one side starts with 100% agreement, they can only tie or lose.

  • If there are no undecided, then the side with less initial agreement can be viewed as having a larger sample size from which to draw.

  • The undecided side is not likely to be truly undecided. If we assume that the two sides are equally polarized, it seems our prior belief about the undecided population should be $\text{Beta}(\text{# For}, \text{# Against})$ if each was forced to take a side.

Given that we have to rely on audience polling, is there a more fair way to judge who wins?

$\endgroup$
  • 1
    $\begingroup$ I'd think something like the "For-Against ratio -After" divided by the "For-Against ratio -Before" (essentially an odds ratio) would be a better choice. If it's higher than 1 you improved the odds, if it's less than 1, you didn't. $\endgroup$ – Glen_b -Reinstate Monica Apr 18 '14 at 0:02
  • $\begingroup$ That was my initial thought too, though I formulated it as percentage gain. I'm just not sure how to prove that it's an unbiased estimate. $\endgroup$ – Wesley Tansey Apr 19 '14 at 16:41
  • $\begingroup$ An unbiased estimate of what? I'm not sure unbiasedness is an especially desirable property for this. $\endgroup$ – Glen_b -Reinstate Monica Apr 19 '14 at 16:47
  • $\begingroup$ Of how well each side did. Ideally we would not want to bias the outcome based on the initial response of the crowd. Or I may be thinking about this completely wrong... $\endgroup$ – Wesley Tansey Apr 19 '14 at 19:28
  • $\begingroup$ Ah, I think we're using bias in a slightly different way there. Whether my suggestion is biased in that sense depends on what exactly you're trying to measure. By one popular measure, it perfectly deals with that issue. $\endgroup$ – Glen_b -Reinstate Monica Apr 20 '14 at 0:10
10
+50
$\begingroup$

Your concerns are well founded. Unfortunately, there are many defensible, objective ways to settle this issue and they can conflict with one another. The following analysis provides a framework for deciding how you might want to evaluate the outcome and shows how dependent your conclusions are on assumptions you make about the dynamics of the situation.


We have little or no control over the initial audience. It might not represent a larger population (such as all viewers) in which we are more interested. Therefore, absolute numbers of opinions are of little relevance: what matters are the rates at which people might change their minds. (From these rates we could estimate how the listening population might change, given information about their initial opinions, even when the proportions of opinions in the listening audience differ from the studio audience that was polled.)

The outcome therefore consists of six possible changes of opinion and six associated rates of change:

  • Those "for," whom I will index with $1,$ can change their mind and end up either against (with index $2$) at rate $a_{12}$ or undecided (with index $3$) at rate $a_{13}$.

  • Those "against" can change their mind to "for" at rate $a_{21}$ or "undecided" at rate $a_{23}$.

  • The undecideds can change their minds to "for" at rate $a_{31}$ or "against" at rate $a_{32}.$

Define $a_{ii}$, for $i=1,2,3,$ to be the proportion of people of index $i$ not changing their minds.

The columns of the matrix $\mathbb{A}=(a_{ij})$ contain nonnegative numbers which must add to unity (assuming everybody who responds to the initial poll also responds to the final one). That leaves six independent values to determine based on the transition from the initial distribution in the audience, $x=(0.18, 0.42, 0.40)$, to the final distribution $y=(0.23, 0.49, 0.28) = \mathbb{A}x$. This is an underdetermined system of (constrained) linear equations, leaving tremendous flexibility in deriving a solution. Let's look at three solutions.

Solution 1: Least Change

We might ask the transition matrix $\mathbb{A}$ to be as small as possible in some sense. One way is to minimize the total proportions of people who change their opinions. This is accomplished in the example with the solution

$$\mathbb{A}=\left( \begin{array}{ccc} 1 & 0 & 0.125 \\ 0 & 1 & 0.175 \\ 0 & 0 & 0.700 \\ \end{array} \right).$$

That is, $12.5\%$ of the undecideds ended up for, $17.5\%$ of them ended up against, and none of the original fors or againsts changed their minds. Who won? The againsts, obviously, because the debate persuaded a larger proportion of the undecideds to settle for the "against" opinion.

This model would be appropriate when you believe the initial factions are hardened to their opinions and the only people likely to change their minds are among those initially declared as undecided.

Solution 2: Least Squares

A mathematically simple solution is to find the matrix $\mathbb{A}$ whose squared $L^2$ norm $||\mathbb{A}||_2^2 = tr(\mathbb{A}^\prime \mathbb{A})$ is as small as possible: this minimizes the sum of squares of all nine transition probabilities (which include the $a_{ii}$ representing the proportions who do not change their minds). Its solution (rounded to two decimal places) is

$$\mathbb{A} = \left( \begin{array}{ccc} 0.28 & 0.22 & 0.22 \\ 0.41 & 0.51 & 0.50 \\ 0.31 & 0.27 & 0.28 \\ \end{array} \right).$$

Comparing the rows, we see that although $22\%$ of the "against" side was persuaded to convert to "for" (and another $27\%$ were sufficiently confused to become undecided), fully $41\%$ of the "for" side was converted (and another $31\%$ were confused). The original undecideds tended to convert to the "against" side ($50\%$ versus $22\%$). Now "against" is the clear winner.

The least squares solution typically posits a lot of change in each group. (Subject to the constraints of the problem, it is trying to make the changes all equal to $1/3$.) Whether it corresponds to a realistic portrayal of the population is difficult to determine, but it does exhibit a mathematically possible picture of what happened during the debate.

Solution 3: Penalized Least Squares

To control and limit the rate at which people change their opinions, let's penalize the least squares objective by including terms that favor no change of opinion. These are the terms on the diagonal of $\mathbb{A}$. We might suppose that it's harder to change the opinion of somebody who is not undecided, so it would be good to downweight the latter. To this end introduce positive weights $\omega_i$ and find $\mathbb{A}$ for which $$||\mathbb{A}||_2^2 - \omega_1 a_{11} - \omega_2 a_{22} - \omega_3 a_{33}$$ is minimized.

For example, let's downweight the undecideds by 50% by selecting weights $\omega = (1,1,1/2)$. The (rounded) solution is

$$\mathbb{A} = \left( \begin{array}{ccc} 0.91 & 0 & 0.17 \\ 0.03 & 0.93 & 0.23 \\ 0.06 & 0.07 & 0.60 \\ \end{array} \right).$$

This solution is intermediate between the first two: a small proportion of the committed sides changed their minds or became undecided while $40\%$ of the undecideds made a decision ($17\%$ for and $23\%$ against). Once again, however, the results clearly favor the "against" faction.

Summary

In this transition model of opinion change, most solution methods indicate a win for the "against" side in this particular example. Absent any strong opinions about the dynamics of change, that suggest the "against" side won.

In other circumstances some solution methods might indicate one winner and other solution methods another winner. For instance, in the transition from $(.20,.60,.20)$ to $(.30,.40,.30)$ it naively looks like the "fors" had a spectacular win: their numbers increased from $20\%$ to $30\%$ while the "against" faction decreased from $40\%$ to $30\%$. However, the (rounded) least squares solution at least suggests there is a way this could happen in which the debate slightly favored the other side! It is

$$\mathbb{A} = \left( \begin{array}{ccc} 0.32 & 0.29 & 0.32 \\ 0.36 & 0.42 & 0.36 \\ 0.32 & 0.29 & 0.32 \\ \end{array} \right).$$

Here, $36\%$ of the "fors" changed to the other side while only $29\%$ of the "against" changed to the opposite opinion. Moreover, slightly more of the undecideds $(36\%)$ vs $32\%$) came out "against" rather than for. Although their numbers in this audience decreased, we have a situation (reminiscent of Simpson's Paradox) in which the "against" faction clearly won the debate!

Additional Comments

If the opinion polls could track individuals both before and after, we could estimate the entire transition matrix $\mathbb{A}$ and there would be far less uncertainty about the effects of the debate on public opinion.

The three solution methods illustrated here are not the only possible ones: others could be found by weighting the coefficients of $\mathbb{A}$ individually, for instance. They do cover a wide gamut of possibilities, though, ranging from the parsimonious "least change" solution through the aggressive least squares solution. Thus, exploring the range of solutions obtained with these three methods should give a good indication of what might reasonably be achieved. If they all agree on the outcome, it should be in little doubt.

$\endgroup$
  • $\begingroup$ Thanks for the detailed post! I'm concerned though that all of these methods do not consider the possibility that the undecideds are not truly undecided. $\endgroup$ – Wesley Tansey Apr 22 '14 at 19:50
  • $\begingroup$ They have the flexibility to incorporate your concern about that possibility. You're still stuck with the need to make (strong) assumptions: if you think they're not truly decided, you will have to estimate which proportion is "for" and which proportion "against" (and it would be folly to assume the proportions are the same as the number for:number against!) One way to sidestep such estimation--if only to see what the result might look like--is to choose a solution that rewards a change of opinion by an undecided person. $\endgroup$ – whuber Apr 22 '14 at 20:02
  • $\begingroup$ Assuming both sides are equally polarizing, wouldn't your MAP estimate of the undecided people be the for:against ratio? $\endgroup$ – Wesley Tansey Apr 22 '14 at 21:08
  • $\begingroup$ In most circumstances it would be difficult to support such an assumption. For instance, less-informed people might have a greater tendency to be undecided--and also have a greater tendency eventually to favor one of the two positions. The effect of an "equally polarizing" assumption could be so strong (especially when there is a large proportion of undecideds) as to render subsequent analysis beside the point: the results would primarily be a consequence of that assumption. A productive line of thought for you might be to consider gathering additional information about the undecided people. $\endgroup$ – whuber Apr 22 '14 at 22:28
3
$\begingroup$

The issue of bias here seems to be that one side may be favored to win even if they don't have better debating skill, rather than the statistical concept of bias of an estimator. A natural approach would be to tackle this concern directly: use data from previous contests to fit a regression model \begin{equation} p(\textrm{for}_\textrm{after},\textrm{against}_{\textrm{after}},\textrm{undecided}_{\textrm{after}} \mid \textrm{for}_\textrm{before},\textrm{against}_{\textrm{before}},\textrm{undecided}_{\textrm{before}}) \end{equation} and set the winning rule as a function of the before-debate-poll so that the predictive probability of winning is $0.5$ for both teams. Note that there are still multiple choices for the decision rule as the outcome space is 2-dimensional but, if we trust the predictive model, this doesn't matter in terms of fairness of the contest. One could, e.g., just decide that the for-team wins if the For-Against ratio after the debate exceeds its predictive median (conditional on the before-poll).

Ideas for building a predictive model

Initially I had in mind just some "black-box" model of the after-poll-numbers as a function of the before-poll-numbers and noise. However, a better approach might be to borrow whuber's idea of considering the transition probabilities. Simplest (though maybe not realistic) approach would be to consider the transition probabilities as independent of the before-debate-poll numbers. For example, assume the transition probabilities are drawn from Dirichlet distributions: \begin{align} (P(\textrm{for} \mid \textrm{for before}),P(\textrm{ud} \mid \textrm{for before}),P(\textrm{ag} \mid \textrm{for before})) & \sim Dir(a_{ff},a_{uf},a_{af}) \\ (P(\textrm{for} \mid \textrm{ud before}),P(\textrm{ud} \mid \textrm{ud before}),P(\textrm{ag} \mid \textrm{ud before})) & \sim Dir(a_{fu},a_{uu},a_{au}) \\ (P(\textrm{for} \mid \textrm{ag before}),P(\textrm{ud} \mid \textrm{ag before}),P(\textrm{ag} \mid \textrm{ag before})) & \sim Dir(a_{fa},a_{ua},a_{aa}), \end{align} where the $P$s are transition probabilities for individuals and the $a$s are hyperparameters that control how the transition probabilities vary from debate to another. The $a$s are learned from data of previous shows, either by optimizing point estimates (e.g. maximum a posteriori or maximum likelihood) these, or a full Bayesian solution that outputs a posterior distribtuion of the $a$s. One could also add some symmetry constraints if one wants to assume for and against behave similarly (before knowing the particular debate question) e.g., $a_{ff}=a_{aa}$, $a_{fu}=a_{au}$.

Given posterior distributions or point estimates of $a$s, and the distribution of individuals in current before poll (that I now assumed to be independent of the transition probabilities), it is straightforward to simulate the distribution of after-debate-poll numbers, and then pick the median of, e.g., for/against-ratio as the winning threshold.

$\endgroup$
  • $\begingroup$ Could you expand on the idea of a predictive model with an example? $\endgroup$ – Wesley Tansey Apr 22 '14 at 19:55
  • $\begingroup$ @WesleyTansey I realized one could use whuber's idea of considering the transition probabilities to construct a predictive model for the purposes of my answer. I edited my answer to contain some initial ideas, but I have not tried implementing this nor am I currently planning to. $\endgroup$ – Juho Kokkala Apr 26 '14 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.