12
$\begingroup$

Background:
I had to perform a data analysis for a client (some kind of lawyer) who was an absolute beginner in statistics. He asked me what the term "statistical significance" means and I really tried to explain it... but since I'm not good at explaining things I failed ;)

Improve this question
$\endgroup$

7 Answers 7

Reset to default
17
$\begingroup$

Differences happen as a result of chance.

When we believe something is statistically significant we believe the difference is larger than can reasonably be explained as a chance occurrence.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I like the use of chance, but think that is highly misleading in terms of how significance testing is commonly used. For example, large sample size means you will almost always get significance due to "chance" baseline differences. It is pretty widely accepted that those results can be called "statistically significant" despite being explained reasonably by chance. $\endgroup$
    – Flask
    Dec 15, 2013 at 1:06
  • $\begingroup$ @Flask: in what sense are these baseline differences due to chance? $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Dec 15, 2013 at 3:46
  • $\begingroup$ @Scortchi If randomization has been performed then differences could be due to chance. See here. Even if it has been performed something may introduce bias later. See here. If randomization has not been performed then it can be due to chance or investigator bias or any number of reasons. $\endgroup$
    – Flask
    Dec 15, 2013 at 3:55
  • 1
    $\begingroup$ Useful answer except that it applies only to tests of differences. $\endgroup$
    – rolando2
    Dec 15, 2013 at 4:17
  • 2
    $\begingroup$ +1 This is an excellent answer because it avoids the arcana of p-values, probabilities, distributions, null hypotheses, etc., and gets right to the point in a way that is applicable to what most lawyers will be dealing with. That it might not be comprehensive is beside the point: the details and variations can be handled later. If pressed to improve on this, the main change I would make would be to emphasize that beliefs about statistical significance are based on data: that would distinguish this description from, say, theological beliefs. $\endgroup$
    – whuber ♦
    Dec 15, 2013 at 16:06
4
$\begingroup$

NOTE: what I want to stress in this answer is that statistical significance is a useful tool, but also different from truth.

Take a pack of 52 cards. If my client is innocent it is a normal pack of cards, 13 hearts. If my client is lying it is a fixed pack and all 52 cards are hearts.

I draw the first card and it is a heart. Aha, guilty! Well, obviously common-sense tells us that is not the case: there was a one in four chance this would happen even if he was innocent. We don't have statistical significance just from looking at one card.

So we draw a second card. Another heart. Hhhmmm... definitely guilty then! Well, there were still 12 hearts in those remaining 51 cards, so not impossible. The maths (13/52 * 12/51 = 0.0588) tells us this happens about 6% of the time even if innocent. For most scientists this would still not count.

Draw a third card, another heart! Three in a row. The chances of this happening are (13/52 * 12/51 * 11/50 = 0.01294), so just over 1% of the time this can happen by chance.

In much of science 5% is used as a cut-off point. So if you have no other evidence than those three cards you have a statistically significant result that he is guilty.

The important point is that the more cards you are allowed to look at the better your confidence in his guilt, which is another way of saying the higher the statistic significance becomes.

NOTE: you never have a proof of his guilt unless you are allowed to look at 14 cards. With a normal pack of cards it is theoretically possibly to draw 13 hearts in a row, but 14 is impossible. [Aside for pedants: let's assume the numbers on the cards are not visible; all cards are one of four possible suits, and that is it.]

NOTE: you have proof of his innocence the moment you draw any card other than a heart. This is because there were only two possible packs: normal or all hearts. Real-life is more complicated, and the maths gets more complicated too.

By the way, if your client is not a card player, try Monopoly: everyone rolls a double-six some of the time; but if someone rolls double-six every time you get suspicious. Statistics just allows us to put an exact number on how suspicious we should be.

Improve this answer
$\endgroup$
3
$\begingroup$

My own advice is not to talk about the following things:

  1. p-values,
  2. test-statistics,
  3. the likelihood of things happening by chance alone.

Don't be too hard on yourself about the lawyer. This is an educated person who spent at least a semester in a university Statistics class, and not a bit of it stuck with him. It's the same story for virtually every other non-scientist I've worked with - statistical significance doesn't stick. It's just too unnatural a concept.

I encourage you to explain statistical significance in terms of evidence. Classical statisticians have encoded evidence on a 0 to 1 scale, where smaller values constitute more evidence and 0.05 is where the line is conventionally drawn.

Improve this answer
$\endgroup$
4
  • $\begingroup$ imo the idea of sig. can stick with non-scientists; what's often seen as unnatural is the technical definition, if people get that far. As far as evidence, of course this is about evidence: the question is how one deals with evidence statistically in order to arrive at a decision about sig. $\endgroup$
    – rolando2
    Dec 15, 2013 at 4:16
  • $\begingroup$ I like your optimism, but I disagree that it's obvious to the typical person that statistical significance is all about evidence. I think they see it as some sort of switch at flips when your data set gets large enough and all the numbers computed are now somehow "valid." You're asserting that it's important for the layperson to know how the evidence is quantified, so get ready to talk about probabilities computed under a hypothesis that you probably didn't believe was true in the first place. $\endgroup$
    – Ben Ogorek
    Dec 15, 2013 at 4:31
  • $\begingroup$ Ah but if you talk about evidence, you enter Bayesian land. $\endgroup$
    – Arthur B.
    Dec 20, 2013 at 13:38
  • 1
    $\begingroup$ I don't think Bayesians own "evidence" (the concept), though they have certainly formalized it. I would argue that a small p-value is evidence of something. $\endgroup$
    – Ben Ogorek
    Dec 30, 2013 at 13:42
2
$\begingroup$

"Statistically significant" means that something could have just happened randomly, but it is unlikely. Instead, there is much more likely that there is some kind of cause. You should make this more concrete with an example that is relevant to your client, since that explanation is so abstract.

For example, if the lawyer Anne won many more cases on average than Bill, this could have just happened randomly. However, if Anne won a statistically significant more cases then it is much more likely that there is something that could help to explain why Anne has won more cases than Bill. We don't know the cause. Perhaps Anne is a better lawyer or Bill purposely chooses cases that are more difficult.

Improve this answer
$\endgroup$
0
$\begingroup$

Keep it simple and concise!

A p-value is defined as the probability of getting results as or more extreme as the one we observed assuming the null is true. If the p-value is small enough, the null is likely not true. We arbitrarily choose a cut-off for what we consider to be a "small enough" (alpha) and for all p-values that fall below alpha, we reject the null.

That's how I explain it to my intro stats class.

Improve this answer
$\endgroup$
1
  • $\begingroup$ But what if you have no way of picking a plausible null hypothesis (i.e., no two groups of people are ever exactly equal, but you also do not have enough information to predict anything better than mean1=mean2)? Explaining statistical significance without mentioning the limitations can do harm. $\endgroup$
    – Flask
    Dec 15, 2013 at 2:58
0
$\begingroup$

I'll try.

First you calculate a p-value based on the average data and how variable the data is. The more variable, the less likely to get a small p-value. On the other hand, if, for example, you are comparing two groups, the greater the difference between the averages of them, the smaller the p-value.

Also, the variability of the data can be somewhat canceled out by having more data. Imaging two sets of data with the same difference between two averages and same amount of variability. In this case, the set with larger sample size will have smaller p-value.

The test part is just seeing if the p-value is lower than some number. Usually people use .05, but this is arbitrary social custom. A lot of people think it makes no sense to use an arbitrary number, yet it is very common for historical reasons.

Also keep in mind that just because your significance test says there is a difference between two groups does not mean you know why there is that difference. On the other hand, if the test says there is no significant difference, this could just be because your variability was too large and you didn't have enough data to get a low p value, it does not mean there is no actual difference.

Edit:

To summarize, lower p value means more evidence against the prediction:

Difference from predicted result -> Down p-value

More data -> Down p-value

More variability -> Up p-value

Down p-value means more evidence saying the prediction is false. Every prediction in history has been shown false to some decimal place.

Improve this answer
$\endgroup$
0
$\begingroup$

Statistical significance is a concept used to provide justification for accepting or rejecting a given hypothesis. Given a set of data an analyst can compute statistics and determine the magnitude of various relationships between different variables.

The job of statistics is to determine if the data contains enough evidence to enable you to conclude that the computed statistics or relationships observed between variables can be interpreted as being true statements or if the results observed in your sample data are simply due to chance. This is done by determining some sample statistic that would exhibit certain characteristics if the null hypothesis is true but not if the null hypothesis is false. The more the relevant sample statistic appears to exhibit the characteristics expected under the null hypothesis, the stronger the statistical evidence that the null hypothesis is correct. Likewise, the less the sample statistic appear to exhibit the characteristics expected under the null hypothesis, the weaker the statistical evidence that the null hypothesis is correct.

The amount that the sample statistic exhibits the characteristics expected under the null is a matter of degree, but in order to conclude that the null hypothesis is accepted or rejected there must be some arbitrary cutoff. As such, a cutoff value is chosen. If the sample statistic falls within or on one side of the cutoff value then it is said to conform with the characteristics expected under the null hypothesis, and thus the result can be considered statistically significant for the given cutoff value (e.g. at the 5% alpha level). If the relevant sample statistic falls on the other side of the cutoff value then it is said to not conform with the characteristics expected under the null hypothesis, and thus the result is not considered statistically significant for the given cutoff value.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But how often is there really a distinct population determined beforehand that the results are supposed to apply to. Usually an argument is made to apply the result beyond the exact population studied, which was a unique sample. How much this uniqueness of the subjects/whatever matters is unknown in many circumstances. An exception could be manufacturing quality control, but significance testing is used much more widely than that. I only mean to emphasize the limitations of the procedure, which were omitted from my own education. $\endgroup$
    – Flask
    Dec 15, 2013 at 4:04
  • $\begingroup$ @Flask that's a good point. I've edited my answer to try and make it more general. $\endgroup$
    – tjnel
    Dec 15, 2013 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.