# What does more than chance mean?

I'm interested in what exactly sentences like this mean:

In the former case, participants correctly chose the larger city in 55% of the pairs, in the latter in 54% of the pairs, both rates significantly different from chance.

Is it basically just that its significantly different from 50%?

UPDATE:

Here's another one:

But now here's another one I've found: In a pretest, one brand had been rated as higher quality, and participants could identify the higher-quality product 59 percent of the time in a blind test (substantially higher than chance, which was 33 percent). I still don't get how is 33 determined?


How do they get 33?

Yes, that is probably what was intended, given the phrases "of the pairs".

Of course statistically significant, especially with a large enough sample, does not need to mean substantial.

The context seems to be described here, including

Imagine you are confronted with the names of two German cities, and you recognize neither of them. There is no more information available to infer which city is larger. What can you do?

...

Drösemeyer (2000) and Hell used the names of smaller German cities, between 45.000 and 60.000 and between 20.000 and 25.000 inhabitants. This resulted in their German participants sometimes having heard of neither of the two, and sometimes just recognizing the two by name, but with no further information available from memory. In the former case, participants correctly chose the larger city in 55% of the pairs, in the latter in 54% of the pairs, both rates significantly different from chance. Although these effects are tiny, they point to an interesting exploitation of information from the mere names, even if both cities are unrecognized.

That makes it clear that guessing one out of two is what was involved and suggests that there is on average a tiny positive amount of information about cities' sizes in their names. But obviously it is not a particularly reliable way of estimating city size.

• so they are comparing 54% to 50% ? how can that be meaningful at all? – Dbr Sep 6 '11 at 21:24
• We're assuming that "chance" means 50%. In many cases that's not so--consider an experiment in which more than two choices are possible, for instance. Generally, "chance" refers to the probabilities assigned by the null hypothesis. – whuber Sep 6 '11 at 21:25
• @Daniel What is not meaningful about comparing 54% to 50%? Perhaps you are asking whether the difference is scientifically important. That's a different question than statistical significance. Its answer depends on the context. It might help to think of a "significant" difference as being a detectable one. – whuber Sep 6 '11 at 21:27
• @Daniel It means replacing aspects of the real world with idealized random processes. In my preceding comment I treated the answers as if they had been obtained by drawing values out of a well-mixed box containing equal amounts of zeros and ones. That's a probability model. It describes a situation where respondents are independently guessing the answers to the questions. A more complex model would be needed when it's likely answers will be correlated. E.g., two questions might ask the same thing, so a consistent guesser would give identical answers. – whuber Sep 7 '11 at 18:03
• @Daniel Mathematically, you give a probability distribution. Coding the answers as 0/1 identifies the sample space with $\{0,1\}^6$. A sufficient statistic is the sum of these answers, a 6D vector $(x_i)$ with components in $\{0,1,\ldots,15\}$. Assuming independence among people (which is clear) and among questions (debatable but reasonable in many cases), the null distribution is given by $6$ Binomial$(15,1/2)$ distributions, assigning a probability of $2^{-15\times 6}\prod_{i=1}^6\binom{15}{x_i}$ to $x$. – whuber Sep 7 '11 at 20:38