6
$\begingroup$

Edit: Note that this question is not about multiple unreliable witnesses to the same incident, but rather multiple incidents with only one witness each. Should the accumulation of separate alleged incidents change our level of doubt about any individual alleged incident?

Here's a question about the informal statistical reasoning that many people do about crime and false accusations. I'd like to know whether the "common sense" response makes mathematical sense.

It's the middle of Prohibition. A prominent politician is accused of drinking copious amounts of alcohol during a party. He says that he has only ever consumed alcohol medicinally after strict advice from his doctor. He says that these are scurrilous, completely false accusations by his political enemies.

Most people say, "It's no more reliable than a rumour; he has lots of political enemies who'd have reason to falsely accuse him; much doubt remains."

Then a number of other people make the same accusation about him; different times, different places, different parties, but always lots of drinking.

After these reports, most people say, "With just one accuser, we were doubtful; now, with many additional accusations, there's little doubt that the first accusation was true."

Is this common informal statistical reasoning sound?

If you're approaching the question with Bayesian reasoning, keep in mind that the question is not "do the additional accusations change the likelihood that the first accusation is false?", but rather, "should the additional accusations cause a large change our level of doubt about the first accusation?"

Now let's add a series of prior assumptions. Do any of them change the soundness of the informal statistical reasoning?

  1. The amount of drinking is highly skewed, with a few people drinking the majority of alcohol. (This assumption is still true today, as it happens, though the skew is probably less extreme than it was during the height of Prohibition.)

Would that assumption make any difference to the way that our doubt should change as new accusations emerge?

  1. The vast majority of people - over 99%, let's say - are honest, and would never make a false accusation like this. (Note that most of our informal reasoners are wrong about this; they assume much more dishonesty than actually exists.)

Would that make any difference?

And finally:

  1. The distribution of accusers is also skewed: Most people who could make true accusations keep their mouth shut, since admitting they were at a party would damage their own reputation; most people who do make accusation make true accusations; a very small number of people make repeated false accusations. However, we have no evidence to determine which group the original accuser falls into.

Would that make a difference? If it would, what effect would the amount of skew have?

If these factors make a difference, which one makes the biggest difference?

I eagerly await input from the finest minds of Stack Exchange. Again, remember that the question is whether our level of doubt about the initial accusation should change a little, a lot, or not at all after more accusations surface.

$\endgroup$
  • $\begingroup$ It seems to me the thrust of these questions is not terribly different than the little exercise recently proposed at stats.stackexchange.com/questions/117033. Certainly the solution--which is simple to obtain--provides a sufficient tool for a quantitative analysis of your three sub-questions. $\endgroup$ – whuber Nov 3 '14 at 23:11
  • 1
    $\begingroup$ @whuber: That exercise appears to be about multiple unreliable witnesses to the same incident. This one is somewhat different, since the witnesses are all describing different incidents. I hope that makes the question more interesting, if somewhat more difficult. $\endgroup$ – Andrew Klaassen Nov 3 '14 at 23:50
  • 2
    $\begingroup$ Would you mind explaining or posting a link to an explanation of the distinction you are making here: "do the additional accusations change the likelihood that the first accusation is false?", but rather, "should the additional accusations cause a large change our level of doubt about the first accusation?" $\endgroup$ – Bill Nov 4 '14 at 15:51
  • $\begingroup$ @Bill: It has been explained to me a few times that additional evidence doesn't change the "likelihood" of anything; it either did happen or it didn't. My distinction is an attempt to avoid that detour. $\endgroup$ – Andrew Klaassen Nov 4 '14 at 17:11
1
+50
$\begingroup$

An approach to the matter may be as follows:

There exists a Bernouli random variable $Y_i$, which models whether the politician drunk ($Y_i=1$) or not ($Y_i=0$) during a specific party, indexed by $i$. Since nothing is really unprobable under the moon (parties being held usually during night hours), there exists some strictly positive probability $p_i>0$ that the politician did drink during party $i$ (this is also consistent with Cromwell's rule). Since parties can only be attended sequentially, the collection of these random variables forms a stochastic process through time, $\{Y_i\}_{i\in N}$. Note that it is difficult to argue that the elements of the process are independent, since after all we are talking about the same person.

This process is not observed by the general public. Instead, what we observe is the process of public statements made by participants in the same parties as to whether the politician did drink or not. Denote this process $\{X_i\}_{i\in N}$. We assume that no conflicting statements are made for the same party. So either somebody declares that the politician drunk, or $X$ takes the value $0$.

The deterministic question is: Does $X_i=1 \Rightarrow ? \;Y_i=1$. But in a statistical framework, we can only relativize the issue and transform it into the question : Does $X_i=1 \Rightarrow ? \;p_{i} > p_{i-1}$.

We can view the $X$-process as an "imperfect measurement" of the $Y$-process -and the crucial point here is how we evaluate the measurement error. The various "prior assumptions" the OP states at the second part of the question pertain to the evaluation of the reliability of the sample. Allowing for conflicting statements that negate the previous ones, again, affects how we will evaluate the reliability of the sample.

Assuming some reliability, accumulation of realizations of the $X$-process where $X_i=1$, is bound to increase the probability that the politician is/has become a drinker, and this can be mapped to the question whether a structural break had happened, given the sample: after all people may change as time passes, and so the politician may have attended many parties without drinking, but at some point things changed. But when did this happen?

So it is not false reasoning to argue that these signals may also make us revise any assessments we have made in the past about the matter, because, it has to do with when the structural break actually happened -and it may be the case that it did happen the first time it was declared that it did, even though, back then, we did not concede as much, due to thin evidence.

So, in this framework, this "common informal statistical reasoning" is not invalid.

...And real life agrees: say, fraud is alleged. Again. And Again. Finally, auditors appear. Why will they go over past transactions, as they do? Because they reckon "the fact that signals of fraud got sufficiently strong in order for us to investigate only now (given limited resources, materiality principle and the like), does not mean that fraud was not conducted in the past. Including the specific past instances where fraud was specifically alleged in the past". Have editors fallen into some logical or statistical fallacy? I wouldn't say so, if experience from audit results is any indication.

$\endgroup$
  • $\begingroup$ Would it be fair to say that an assumption you've added which may make a bigger difference than the ones I've suggested is an assumption about the consistency of human behaviour? Still chewing through the rest of your answer. $\endgroup$ – Andrew Klaassen Nov 7 '14 at 18:25
  • $\begingroup$ You mean the possible intertemporal _in_consistency of human behavior (which is just a fancy way to say "change")? $\endgroup$ – Alecos Papadopoulos Nov 7 '14 at 18:28
  • $\begingroup$ I'm thinking of the entire spectrum of assumptions that one could make about behavioural consistency: Completely random behaviour, with each incident unconnected to any other; completely consistent behaviour, with no change; the position you've taken between the two (but closer to the second), with consistent behaviour interrupted by a "structural break"; or some other way for change and consistency to mix. Would I be right to guess that the correctness of our "common informal reasoning" will vary with how close human behaviour is to being completely consistent? $\endgroup$ – Andrew Klaassen Nov 7 '14 at 20:11
  • $\begingroup$ As long as behavior is not "completely consistent", the reasoning is valid -its "correctness" is a matter of evaluating the reliability of the sample. "Complete consistency" would make the whole set up uninteresting. Under it, If we cannot observe the politician at all, then we will have to form a prior belief which, in order to be consistent with the "completely consistent behavior", it can only assign "zero or one" probabilities, violating Cromwell's rule and making any "evidence" down the road irrelevant. $\endgroup$ – Alecos Papadopoulos Nov 7 '14 at 20:18
  • 1
    $\begingroup$ Too broad, I think. $\endgroup$ – Alecos Papadopoulos Nov 7 '14 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.