I think I understand the concept of p-value but unfortunately I still have to exert a lot of brain cycles to get my arms around it.
I would like to get an explanation of the p-value that is rigorous enough for a sophisticated layman - something that would be intuitive.
Take a look at the tooth brushing example at the very start of Chapter 14 of Andrew Vickers' book What is a p-value anyway? 34 Stories to Help You Actually Understand Statistics. It starts on page 57 or you can use the table of contents button in the bottom left corner to find it.
Here's an excerpt:
[I]f you do nothing else, please try to remember the following sentence: “the $p$-value is the probability that the data would be at least as extreme as those observed, if the null hypothesis were true.” Though I’d prefer that you also understood it—about which, teeth brushing.
I have three young children. In the evening, before we get to bedtime stories (bedtime stories being a nice way to end the day), we have to persuade them all to bathe, use the toilet, clean their teeth, change into pajamas, get their clothes ready for the next day and then actually get into bed (the persuading part being a nice way to go crazy). My five-year-old can often be found sitting on his bed, fully dressed, claiming to have clean teeth. The give-away is the bone dry toothbrush: he says that he has brushed his teeth, I tell him that he couldn’t have.
My reasoning here goes like this: the toothbrush is dry; it is unlikely that the toothbrush would be dry if my son had cleaned his teeth; therefore he hasn’t cleaned his teeth. Or using statistician-speak: here are the data (a dry toothbrush); here is a hypothesis (my son has cleaned his teeth); the data would be unusual if the hypothesis were true, therefore we should reject the hypothesis.
[...]
So here is what to parrot when we run into each other at a bar and I still haven’t managed to work out any new party tricks: “The $p$-value is the probability that the data would be at least as extreme as those observed, if the null hypothesis were true.” When I recover from shock, you can explain it to me in terms of a toothbrush (“The probability of the toothbrush being dry if you’ve just cleaned your teeth”).
The other thing I really like about this example is that it also explains that failing to reject the null does not mean the null is necessarily true. Vickers writes that his son has now worked out the trick and has taken to running his toothbrush under the tap for a second or two before heading to bed. Just because the toothbrush is wet (and the data is consistent with the null hypothesis), it does not mean that his son has cleaned his teeth.
