I had to perform a data analysis for a client (some kind of lawyer) who was an absolute beginner in statistics. He asked me what the term "statistical significance" means and I really tried to explain it... but since I'm not good at explaining things I failed ;)
NOTE: what I want to stress in this answer is that statistical significance is a useful tool, but also different from truth.
Take a pack of 52 cards. If my client is innocent it is a normal pack of cards, 13 hearts. If my client is lying it is a fixed pack and all 52 cards are hearts.
I draw the first card and it is a heart. Aha, guilty! Well, obviously common-sense tells us that is not the case: there was a one in four chance this would happen even if he was innocent. We don't have statistical significance just from looking at one card.
So we draw a second card. Another heart. Hhhmmm... definitely guilty then! Well, there were still 12 hearts in those remaining 51 cards, so not impossible. The maths (13/52 * 12/51 = 0.0588) tells us this happens about 6% of the time even if innocent. For most scientists this would still not count.
Draw a third card, another heart! Three in a row. The chances of this happening are (13/52 * 12/51 * 11/50 = 0.01294), so just over 1% of the time this can happen by chance.
In much of science 5% is used as a cut-off point. So if you have no other evidence than those three cards you have a statistically significant result that he is guilty.
The important point is that the more cards you are allowed to look at the better your confidence in his guilt, which is another way of saying the higher the statistic significance becomes.
NOTE: you never have a proof of his guilt unless you are allowed to look at 14 cards. With a normal pack of cards it is theoretically possibly to draw 13 hearts in a row, but 14 is impossible. [Aside for pedants: let's assume the numbers on the cards are not visible; all cards are one of four possible suits, and that is it.]
NOTE: you have proof of his innocence the moment you draw any card other than a heart. This is because there were only two possible packs: normal or all hearts. Real-life is more complicated, and the maths gets more complicated too.
By the way, if your client is not a card player, try Monopoly: everyone rolls a double-six some of the time; but if someone rolls double-six every time you get suspicious. Statistics just allows us to put an exact number on how suspicious we should be.
My own advice is not to talk about the following things:
- the likelihood of things happening by chance alone.
Don't be too hard on yourself about the lawyer. This is an educated person who spent at least a semester in a university Statistics class, and not a bit of it stuck with him. It's the same story for virtually every other non-scientist I've worked with - statistical significance doesn't stick. It's just too unnatural a concept.
I encourage you to explain statistical significance in terms of evidence. Classical statisticians have encoded evidence on a 0 to 1 scale, where smaller values constitute more evidence and 0.05 is where the line is conventionally drawn.
"Statistically significant" means that something could have just happened randomly, but it is unlikely. Instead, there is much more likely that there is some kind of cause. You should make this more concrete with an example that is relevant to your client, since that explanation is so abstract.
For example, if the lawyer Anne won many more cases on average than Bill, this could have just happened randomly. However, if Anne won a statistically significant more cases then it is much more likely that there is something that could help to explain why Anne has won more cases than Bill. We don't know the cause. Perhaps Anne is a better lawyer or Bill purposely chooses cases that are more difficult.
Keep it simple and concise!
A p-value is defined as the probability of getting results as or more extreme as the one we observed assuming the null is true. If the p-value is small enough, the null is likely not true. We arbitrarily choose a cut-off for what we consider to be a "small enough" (alpha) and for all p-values that fall below alpha, we reject the null.
That's how I explain it to my intro stats class.
First you calculate a p-value based on the average data and how variable the data is. The more variable, the less likely to get a small p-value. On the other hand, if, for example, you are comparing two groups, the greater the difference between the averages of them, the smaller the p-value.
Also, the variability of the data can be somewhat canceled out by having more data. Imaging two sets of data with the same difference between two averages and same amount of variability. In this case, the set with larger sample size will have smaller p-value.
The test part is just seeing if the p-value is lower than some number. Usually people use .05, but this is arbitrary social custom. A lot of people think it makes no sense to use an arbitrary number, yet it is very common for historical reasons.
Also keep in mind that just because your significance test says there is a difference between two groups does not mean you know why there is that difference. On the other hand, if the test says there is no significant difference, this could just be because your variability was too large and you didn't have enough data to get a low p value, it does not mean there is no actual difference.
To summarize, lower p value means more evidence against the prediction:
Difference from predicted result -> Down p-value
More data -> Down p-value
More variability -> Up p-value
Down p-value means more evidence saying the prediction is false. Every prediction in history has been shown false to some decimal place.
Statistical significance is a concept used to provide justification for accepting or rejecting a given hypothesis. Given a set of data an analyst can compute statistics and determine the magnitude of various relationships between different variables.
The job of statistics is to determine if the data contains enough evidence to enable you to conclude that the computed statistics or relationships observed between variables can be interpreted as being true statements or if the results observed in your sample data are simply due to chance. This is done by determining some sample statistic that would exhibit certain characteristics if the null hypothesis is true but not if the null hypothesis is false. The more the relevant sample statistic appears to exhibit the characteristics expected under the null hypothesis, the stronger the statistical evidence that the null hypothesis is correct. Likewise, the less the sample statistic appear to exhibit the characteristics expected under the null hypothesis, the weaker the statistical evidence that the null hypothesis is correct.
The amount that the sample statistic exhibits the characteristics expected under the null is a matter of degree, but in order to conclude that the null hypothesis is accepted or rejected there must be some arbitrary cutoff. As such, a cutoff value is chosen. If the sample statistic falls within or on one side of the cutoff value then it is said to conform with the characteristics expected under the null hypothesis, and thus the result can be considered statistically significant for the given cutoff value (e.g. at the 5% alpha level). If the relevant sample statistic falls on the other side of the cutoff value then it is said to not conform with the characteristics expected under the null hypothesis, and thus the result is not considered statistically significant for the given cutoff value.