If 900 out of 1000 people say a car is blue, what is the probability that it is blue?

Question

This initially arose in connection some work we are doing to a model to classify natural text, but I've simplified it... Perhaps too much.

You have a blue car (by some objective scientific measure - it is blue).

You show it to 1000 people.

900 say it is blue. 100 do not.

You give this information to someone who cannot see the car. All they know is that 900 people said it was blue, and 100 did not. You know nothing more about these people (the 1000).

Based on this, you ask the person, "What is the probability that the car is blue?"

This has caused a huge divergence of opinion amongst those I have asked! What is the right answer, if there is one?

Answer 1

TL;DR: Unless you assume people are unreasonably bad at judging car color, or that blue cars are unreasonably rare, the large number of people in your example means the probability that the car is blue is basically 100%.

Matthew Drury already gave the right answer but I'd just like to add to that with some numerical examples, because you chose your numbers such that you actually get pretty similar answers for a wide range of different parameter settings. For example, let's assume, as you said in one of your comments, that the probability that people judge the color of a car correctly is 0.9. That is: $$p(\text{say it's blue}|\text{car is blue})=0.9=1-p(\text{say it isn't blue}|\text{car is blue})$$ and also $$p(\text{say it isn't blue}|\text{car isn't blue})=0.9=1-p(\text{say it is blue}|\text{car isn't blue})$$

Having defined that, the remaining thing we have to decide is: what is the prior probability that the car is blue? Let's pick a very low probability just to see what happens, and say that $p(\text{car is blue})=0.001$, i.e. only 0.1% of all cars are blue. Then the posterior probability that the car is blue can be calculated as:

\begin{align*} &p(\text{car is blue}|\text{answers})\\ &=\frac{p(\text{answers}|\text{car is blue})\,p(\text{car is blue})}{p(\text{answers}|\text{car is blue})\,p(\text{car is blue})+p(\text{answers}|\text{car isn't blue})\,p(\text{car isn't blue})}\\ &=\frac{0.9^{900}\times 0.1^{100}\times0.001}{0.9^{900}\times 0.1^{100}\times0.001+0.1^{900}\times0.9^{100}\times0.999} \end{align*}

If you look at the denominator, it's pretty clear that the second term in that sum will be negligible, since the relative size of the terms in the sum is dominated by the ratio of $0.9^{900}$ to $0.1^{900}$, which is on the order of $10^{58}$. And indeed, if you do this calculation on a computer (taking care to avoid numerical underflow issues) you get an answer that is equal to 1 (within machine precision).

The reason the prior probabilities don't really matter much here is because you have so much evidence for one possibility (the car is blue) versus another. This can be quantified by the likelihood ratio, which we can calculate as: $$ \frac{p(\text{answers}|\text{car is blue})}{p(\text{answers}|\text{car isn't blue})}=\frac{0.9^{900}\times 0.1^{100}}{0.1^{900}\times 0.9^{100}}\approx 10^{763} $$

So before even considering the prior probabilities, the evidence suggests that one option is already astronomically more likely than the other, and for the prior to make any difference, blue cars would have to be unreasonably, stupidly rare (so rare that we would expect to find 0 blue cars on earth).

So what if we change how accurate people are in their descriptions of car color? Of course, we could push this to the extreme and say they get it right only 50% of the time, which is no better than flipping a coin. In this case, the posterior probability that the car is blue is simply equal to the prior probability, because the people's answers told us nothing. But surely people do at least a little better than that, and even if we say that people are accurate only 51% of the time, the likelihood ratio still works out such that it is roughly $10^{13}$ times more likely for the car to be blue.

This is all a result of the rather large numbers you chose in your example. If it had been 9/10 people saying the car was blue, it would have been a very different story, even though the same ratio of people were in one camp vs. the other. Because statistical evidence doesn't depend on this ratio, but rather on the numerical difference between the opposing factions. In fact, in the likelihood ratio (which quantifies the evidence), the 100 people who say the car isn't blue exactly cancel 100 of the 900 people who say it is blue, so it's the same as if you had 800 people all agreeing it was blue. And that's obviously pretty clear evidence.

(Edit: As Silverfish pointed out, the assumptions I made here actually implied that whenever a person describes a non-blue car incorrectly, they will default to saying it's blue. This isn't realistic of course, because they could really say any color, and will say blue only some of the time. This makes no difference to the conclusions though, since the less likely people are to mistake a non-blue car for a blue one, the stronger the evidence that it is blue when they say it is. So if anything, the numbers given above are actually only a lower bound on the pro-blue evidence.)

Answer 2

The correct answer depends on information not specified in the problem, you will have to make some more assumptions to derive a single, definitive answer:

• The prior probability the car is blue, i.e. your belief that the car is blue given you have not yet asked anyone.
• The probability someone tells you the car is blue when it actually is blue, and the probability they tell you the car is blue when it actually is not blue.
• The probability that the car actually is blue when someone says it is, and the probability that the car is not blue, when someone says it is blue.

With these pieces of information, we can break the whole thing down with Bayes's formula to derive a posterior probability that the car is blue. I'll focus on the case where we only ask one person, but the same reasoning can be applied to the case where you ask $1000$ people.

$$\begin{align*} P_{post} (\text{car is blue}) &= P(\text{car is blue} \mid \text{say is blue}) P(\text{say is blue}) \\ & \ \ \ \ + P(\text{car is blue} \mid \text{say is not blue}) P(\text{say is not blue}) \end{align*}$$

We need to continue to further break down $P(\text{say is blue})$, this is where the prior comes in:

$$\begin{align*} P(\text{say is blue}) = \ &P(\text{say is blue} \mid \text{car is blue}) P_{prior}(\text{car is blue}) \\ &+ P(\text{say is blue} \mid \text{car is not blue}) P_{prior}(\text{car is not blue}) \end{align*}$$

So two applications of Bayes's rule get's you there. You'll need to determine the unspecified parameters based on either information you have about the specific situation, or by making some reasonable assumptions.

There are some other combinations of what assumptions you can make, based on:

$$ P(\text{say is blue} \mid \text{car is blue}) P(\text{car is blue}) = P(\text{car is blue} \mid \text{say is blue}) P(\text{say is blue}) $$

At the outset, you don't know any of these things. So you must make some reasonable assumptions about three of them, and then the fourth is determined from there.

Answer 3

There's an important assumption that your 1000 opinions don't share a systematic bias. Which is a reasonable assumption here, but could be important in other cases.

Examples might be:

• they all share a similar colorblindness (genetics in a population for example),
• they all saw the car at night under orange sodium street lighting,
• they all share a common culture in which blue is taboo or magically associated (which biases whether or not they describe any object as blue or use a cultural euphemism or whatever instead),
• they have all been told (or share a common belief) that if they do/don't answer some specific way, something good/bad will happen to them.....

It isn't likely in this case but its a significant implied assumption in other cases. It doesn't have to be that extreme either - transpose your question to some other domain and this will be a real factor.

Examples for each where your answer may be affected by a shared bias:

• ask if a tall thin glass holds more than an actually-identical short fat glass, but your 1000 respondents are very young children (shared misperception).
• ask 1000 people if walking under a ladder is dangerous (common cultural belief)
• ask 1000 married people if they love their partner/have had an affair, in circumstances where they believe their partner will know of their answer. The context might be a TV show, or partner present when asked etc (common belief about consequences)

It wouldn't be hard to imagine some structurally identical questions where the 900:100 response was a measure of beliefs and honesty, or something else, and doesn't point to the correct answer. Not likely in this case but in other cases - yes.

Answer 4

One reason you're getting different answers from different people is that the question can be interpreted in different ways, and it isn't clear what you mean by "probability" here. One way to make sense of the question is to assign priors and reason using Bayes' rule as in Matthew's answer.

Before asking for probabilities, you have to decide what's modeled as random and what's not. It's not universally accepted that unknown but fixed quantities should be assigned priors. Here's a similar experiment to yours that highlights the problem with the question:

Assume $X_i$, $i = 1, \dots, 1000$ are i.i.d. Bernoulli random variables with success probability (mean) $p = 0.5$. For interpretability, let's think of the $X_i$ as coin flips. Suppose you observe (the sufficient statistic) $\sum_{i = 1}^{1000}X_i = 900$. What is the probability that the coin is fair?

From a frequentist perspective the question is either nonsensical or the answer is "one". If you're Bayesian maybe you want to assign a prior distribution to $p$, in which case the question makes sense. The fundamental difference between my example and the question is that $p$ is unknown in the question, and the question disguises the fact that the actual randomness is whether a (presumably randomly sampled) person answers that the car is blue or not. The car's color is not randomly assigned and thus it's uninteresting to speak of the probability of it being blue from a frequentist perspective.

Answer 5

Simple practical answer:

The probability can easily range from 0% to 100% depending on your assumptions

Though I really like the existing answers, in practice it basically boils down to these two simple scenarios:

Scenario 1: People are assumed to be very good at recognizing blue when it is blue ... 0%

In this case, there are so many people stating that the car is not blue, that it is very unlikely that the car is actually blue. Hence, the probability approaches 0%.

Scenario 2: People are assumed to be very good at recognizing not-blue when it is not-blue ... 100%

In this case, there are so many people stating that the car is blue, that it is very likely that it is indeed blue. Hence the probability approaches 100%.

---

Of course coming at this from a mathematical angle you would start by something generic like 'let us assume that the relevant probabilities are ...', which is quite meaningless as such things are typically not known for any random circumstance. Hence I advocate looking at the extremes to grasp the idea that both percentages can easily be justified with simple and realistic assumptions, and that there is therefore no single meaningful answer.

Answer 6

You need to develop some framework of estimation. Some questions you might ask are

1. How many colors are there? Are we talking two colors? Or all the colors of the rainbow?

2. How distinct are the colors? Are we talking blue and orange? Or blue, cyan, and turquoise?

3. What does it mean to be blue? Are cyan and/or turquoise blue? Or just blue itself?

4. How good are these people at estimating color? Are they all graphic designers? Or are they color blind?

From a purely statistical standpoint, we can make some guesses as to the last one. First, we know that at least 10% of the people are choosing an incorrect response. If there are only two colors (from the first question), then we might say that there is

Probability says blue and is blue = 90% say is blue * 90% correct = 81%
Probability says blue and is not = 90% * 10% incorrect = 9%
Probability says not but is blue = 10% * 90% incorrect = 9%
Probability says not and is not = 10% * 10% = 1%

As a quick check, if we add those together, we get 100%. You can see a more mathematical notation of this at the @MatthewDrury answer.

How do we get the 90% in the third? It's how many people said blue but were wrong if it's not. Because there are only two colors, these are symmetric. If there were more than two colors, then the chance of the wrong choice being blue when they said something else would be lower.

Anyway, this method of estimation gives us 90% blue. This includes an 81% chance of people saying blue when it is and a 9% chance of people saying that it isn't when it is. This is probably the closest we can come to answering the original question, and it requires us to rely on the data to estimate two different things. And to assume that the chance of blue being chosen is the same as the chance of blue being correct.

If there are more than two colors, then the logic is going to change a bit. The first two lines stay the same, but we lose the symmetry in the last two lines. In that case, we need more input. We might conceivably estimate the chance of correctly saying blue as 81% again, but we have no idea what the chances are that the color is blue when someone says that it is not.

We could also improve upon even the two color estimate. Given a statistically significant number of cars of each color, we could have a statistically significant number of people view and categorize them. Then we could count how often people are right when they make each color choice and how often they are right for each color choice. Then we could estimate more accurately given people's actual choices.

You might ask how 90% could be wrong. Consider what happens if there are three colors: azure, blue, and sapphire. Someone might reasonably consider all three of these to be blue. But we want more. We want the exact shade. But who remembers the names of the other shades? Many might guess blue because it is the only matching shade they know. And still be wrong when it turns out to be azure.

Answer 7

An exact, mathematical, true/false probability cannot be computed with the information you provide.

However, in real life such information is never available with certainty. Therefore, using our intuition (and where all my money would go if we were betting), the car is definitely blue. (some believe this is not statistics anymore, but well, black/white views of science are not very helpful)

The reasoning is simple. Assume the car is not blue. Then 90% of the people (!) were wrong. They could only be wrong because of a list of issues including:

• color blindness
• pathological lying
• being under the influence of substances like alcohol, LCD, etc
• not understanding the question
• other form of mental disorder
• a combination of the above

Since the above is clearly not likely to affect 90% of an average random population (e.g. colour blindness affects around 8% of males and 0.6% of females, that is 43 people out of 1000), it is necessarily the case that the car is blue. (That is were all my money would go anyway).

Answer 8

I would not eat feces based on the fact that billion of flies can't be wrong. There might dozens of other reasons why 900 people out of 1000 might have been cheated to think the car is blue. After all, that's the base of magical tricks, luring people into thinking something removed from reality. If 900 people out of 1000 see a magician stabbing his/her assistant, they will promptly answer the assistant was stabbed, for how improbable an homicide happened on the stage. A blue light on a reflective car paint, anyone?

Answer 9

The questionee knows too little about how the poll was carried out in order to answer the question accurately. As far as he's concerned, the poll can suffer from several problems:

The people taking the poll could have been biased:

1. The car looked blue because of an optical illusion.

2. The color of the car was for some reason difficult to observe, and the people had for some reason been shown a lot of blue cars before this one, making most of them believe this car was probably blue, too.

3. You had paid them to say that the car is blue.

4. You had someone hypnotize all of them into believing that the car is blue.

5. They had made a pact to lie and sabotage the poll.

There may have been correlations among the people taking the poll because of how they were selected or because they affected each other:

1. You accidentally carried out the poll at a mass meeting for people with the same kind of color blindness.

2. You carried out the poll at kindergartens; the girls were not interested in the car and most of the boys had blue as their favorite color, making them imagine that the car was blue.

3. The first person who was shown the car was drunk and thought it looked blue, shouted "IT IS BLUE", influencing everyone else into thinking that the car was blue.

So while the probability that the car is blue if the poll was completely correctly carried out is extremely high (as explained in Ruben van Bergen's answer), the reliability of the poll may have been compromised which makes the chance that the car is not blue not insignificant. How big the questionee estimates this chance to be ultimately depends on his estimations of how likely it is that circumstances have screwed with the poll and of how good you are at carrying out polls (and how mischievous he thinks you are).

Answer 10

What is the definition of "blue"?

Different cultures and languages have different notions of blue. IIRC, some cultures include green within their notion of blue!

Like any natural language word, you can only assume there is some cultural convention on when (and when not) to call things "blue".

Overall, color in language is surprisingly subjective (link from the comments below, thanks @Count Ibilis)

Answer 11

The likelihood could, depending on more refined preconditions, be several different values, but 99.995% is the one that makes the most sense to me.

We know, by definition, that the car is blue (that's 100%), but it is not well-specified what this actually means (that would bet somewhat philosophical). I will assume something is blue in a sense of can-indeed-be-seen-as-blue.

We also know that 90% of test subjects reported it as blue.

We do not know what was asked or how the evaluation was done, and what lighting conditions the car was in. Being asked to name the color, some subjects might e.g. have said "greenish-blue" due to lighting conditions, and the assessor might not have counted that as "blue". The same people might have replied "yes" if the question had been "Is this blue?". I will assume that you did not intend to maliciously deceive your test subjects.

We know that the incidence of tritanopy is about 0,005% which means that if the car could actually be seen as blue, then 99.995% of the test subjects indeed did see the color as blue. That, however, means that 9.995% of the test subjects did not report blue when they clearly saw blue. They were lying about what they saw. This is close to what your life experience tells you as well: people are not always being honest (but, unless there's a motive, they usually are).

Thus, the non-observing person can assume with overwhelming certitude that the car is blue. That would be 100%

Except... except if the non-observing person herself suffers from tritanopy, in which case she would not see the car as blue even though everybody else (or rather, 90% of them) says so. Here it gets philosophical again: If everybody else heard a tree fall, but I didn't, did it fall?

I daresay that the most reasonable, practical answer would be: If the non-observing person happens to be trianope (0.005% chance), then verifying whether the predicted color and the real color as-seen are the same would yield false. Thus, the likelihood is 99.995% rather than 100%.

Further, as a bonus, since we found out that 9.995% of the test subjects are liars, and it is known that all Cretans are liars, we can conclude that we are not in Crete!

Answer 12

You have a blue car (by some objective scientific measure - it is blue).

...

"What is the probability that the car is blue?"

It is 100% blue.

All they know is that 900 people said it was blue, and 100 did not. You know nothing more about these people (the 1000).

Using these numbers (without any context) is utterly nonsense. It all boils down to personal interpretation of the question. We should not go down this path and use Wittgenstein's: "Wovon man nicht sprechen kann, darüber muss man schweigen."

---

Imagine the following question for comparison:

All they know is that 0 people said it was blue, and 0 did not. 
You know nothing more about these people (the 0).

This is basically the same (information less) problem, but it is much more clear that what we think of the color of the car is mostly (if not completely) circumstantial.

---

In the long run, when we get multiple associated questions, then we are able to start guessing answers to such incomplete questions. This is the same for the tit-for-tat algorithm that doesn't work for a single case, but it does work in the long run. In the same sense Wittgenstein came back from his earlier work with his Principal Investigations. We are able to answer these questions, but we need more information/trials/questions. It is a process.

Answer 13

If we assume the car is blue, then 100 out of 1,000 saying it's not blue implies an extreme sample bias of some kind. Perhaps you were sampling only colour-blind people. If we assume the car is not blue, then the sample bias is even worse. So all we can conclude from the data given is that the sample is very biased, and since we don't know how it was biased, we can't conclude anything about the colour of the car.

Answer 14

There have been some answers. I'm by no means a mathematics guru, but well, here is mine.

There can only be 4 possibilities:

case 1) Persons says car is blue and is correct
case 2) Person says car is blue and is incorrect
case 3) Person says car is not blue and is correct
case 4) Person says car is not blue and is incorrect

From the question, you know that the sum of case 1 and case 4 is 900 people (90%), and the sum of case 2 and case 3 is 100 people (10%). However here is the catch: what you don't know is the the distribution within these 2 case pairs. Maybe the sum of case 1 and 4 completely is made up of case 1 (which means car is blue), or perhaps the whole sum is made up of case 4 (which means car is not blue). Same goes for sum of case 2+3. So... What you need is to come up with some way to predict the distribution within case sums. With no other indication in the question (nowhere does it say people are 80% certain to know their colors or anything like that) there is no way you can come up with a certain, definite answer.

Having said this... I do suspect the expected answer is something along the lines of:

P(Blue) = (case 1 + case 4) * 900 / 1000 = (1/4  + 1/4) * 900 / 1000 = 45 %
P(non-Blue) = (case 2 + case  3) * 100 / 1000 = (1/4 + 1/4) * 100 / 1000 = 5%

where remaining 50% is simply unknown, call it the error margin.

Answer 15

Let $X,Y_1,Y_2,\ldots,Y_{1000} \in \{0,1\}$ denote the true color, and the responses, respectively. "Blue" is coded as a $1$, and vice versa. Assume that $p(x)$ is Bernoulli with parameter $p_x$. Assume that each $Y_i|X=1$ is Bernoull with parameter $p_1$, and assume $Y_i|X=0$ is Bernoulli with parameter $p_0$. Also, pick a prior for the parameters $\theta = (p_x,p_0,p_1)$.

You're looking for $p(\theta,x|y_{1:1000}) \propto p(\theta)p(x|\theta)\prod_{i=1}^{1000}p(y_i|x)$. Formulating it like this highlights the fact that (at least if you're a Bayesian) you need to choose priors for these three parameters. The Bayesian viewpoint is nice because you could take advantage of what you know about how often cars are blue, and what you know about peoples' tendencies to agree with reality.

Also you can generalize this model. For example what if the car changes colors, or if you're looking at a sequence of cars (then you have a sequence $\{x_i\}$), or if people are equipped differently to evaluate car colors ($\{y_i|x\}$ are not identically distributed), or if people are basing their decisions on what other people are saying, etc.

Answer 16

The person that cannot see the car does not know it is scientifically proven to be blue. The probability to he/she that the car is blue is 50/50 (it is blue, or it isn't). Polling other people may influence this person's opinion but it does not change the probability that an unseen car is either blue, or not.

All of the above math determines the probability that your sample set can determine if it is blue.