# The concept of 'proven statistically'

When the news talk about things been 'proven statistically' are they using a well-defined concept of statistics correctly, using it wrong, or just using an oxymoron?

I imagine that a 'statistical proof' is not actually something performed to prove a hypothesis, nor a mathematical proof, but more of a 'statistical test'.

-
There's no proof in statistics except that proofs in mathematical statistics should follow mathematical standards. However, such proofs are irrelevant to your question. In terms of your example, no-one except an idiot or humorist claims that each American family has 2.4 children. At most, the claim is that, on average, families have 2.4 children; that can be checked by sample surveys or another census, but as the number may fluctuate or change systematically a different result doesn't mean the first result was incorrect; nor does the same result mean the first result was correct. – Nick Cox Feb 5 '14 at 14:53

What the news people are talking about is anyone's guess and varies with the newscast. Perhaps most common is that they are giving a one sentence summary of research that requires several pages.

However, your last paragraph is mistaken. Statistically, each family does NOT have 2.4 children. The mean is 2.4 children. This is entirely possible. If you take a random sample of American families (tricky to do, but possible) then you would get an estimate of the mean. However, if you took a census of families, then, if the census really got every family (it doesn't) or, if the people it got are representative of the people it didn't get, with regard to number of children, then you would have proven the fact.

However, not only does the census miss people, the people it misses are different in many ways from the people it gets. The Census Bureau therefore tries to figure out how they are different; thus, again, giving an estimate of the number of kids per family.

But there are things you can prove; if you wanted to know, say, the average number of years that each professor in your department had been teaching, you could get accurate data and come up with an exact mean.

Your penultimate paragraph is also problematic as statistical tests are done precisely to prove hypotheses; more precisely, they are done (in the frequentist framework, anyway) to reject a null hypothesis at a given level of significance.

-
Corrected my question. – Quora Feans Feb 5 '14 at 16:41

I think - as with so many things - it's a combination of a widespread cultural misunderstanding and journalistic attempts at punchy shorthand that turns out to sometimes mislead.

Of course conclusions based on statistical inference isn't proof in any kind of hard sense. It's reliant on assumptions, and even then conclusions (at best) are probabilistic (as we get, say with Bayesian inference), and then with frequentist inference you have to add in the usual error of misinterpretation of p-values as the probability that the null is true. That's without even considering issues like publication or reporting bias

You see similar errors just as much with science reporting more generally and it's just as frustrating.

I don't like the phrase 'statistically proven' myself, as I think it gives the wrong impression. While statistics done well is a powerful tool, the things statistics actually tells us can be surprisingly subtle and the appropriate discussion of the meaning of what is learned and the accompanying qualifications placed on the conclusions are often unsuited to the hype and punchiness of a headline or a hurried few paragraphs squeezed in between the usual celeb gossip.

Indeed, even in the academic journals where those sort of qualifications would seem essential, they are often left aside and instead there appears some formulaic pronouncement (different from research area to research area) that is regarded as 'anointing' the result.

I think there is room for carefully explaining the reasoning going from the results of inference (whether point and interval estimation, hypothesis testing, decision-theoretic calculations or even exploratory construction of a few visual comparisons) to the conclusions they lead to. That reasoning is where the real heart of the matter lies (including where the gaps in reasoning would be laid bare, were they explicit) and we rarely see it laid out.

Besides that, we can keep sounding a note of caution

-
(+1) Can you please review each and every one of my future papers? – Matt Krause Feb 6 '14 at 2:08
You might not like it. More than once my referee's reports have ended up significantly longer than the original paper, as I go through in detail what's wrong and how to fix it (most times only to have the response be to cut out entire sections of paper rather than try to fix the most problematic, but usually most interesting, parts, sadly). One of my students had a similarly detailed report on her work from one of her referees; it was actually quite valuable and led to a much better final product. – Glen_b Feb 6 '14 at 2:36

Empirical knowledge is always probabilistic -- never clearly true or false, but always somewhere in between. Statistical "proof" is a matter of collecting enough data to reduce the probability that a hypothesis is wrong to less than some accepted threshold. And the threshold for "truth" or "correctness" differs from one academic discipline to the next. Sociologists are satisfied with a 95% probability of being right, and sometimes settle for less; quantum physicists demand 99.99999% or better.

-
Welcome to the site, @Kevin Krumwiede. Your last sentence is ambiguous. It seems like you are making a common mistake that conflates a p-value of <.05 (eg) w/ a 95% probability that the null hypothesis is false. – gung Feb 6 '14 at 2:19