How to explain hypothesis testing for teenagers in less than 10 minutes? For over a year now I've been giving a one-hour "a taste for statistics" class. Each time I get a different group of kids coming over, and I give them the class.
The theme of the class is that we run an experiment in which 10 kids (who likes drinking coca-cola) are given two (unmarked) cups, one with coca-cola and one with pepsi. The kids are asked to detect, based on taste and smell, which cup has the coca-cola drink.
I then need to explain to them how to decide if the kids are guessing, or if they (or at least, enough of them) really have the ability to taste the difference. Are 10 out of 10 successes good enough? what about 7 out of 10?
Even after giving this class tens of times (in different variations), I still don't feel I know how to get the concept across in a way that most of the class will get it.
If you have any ideas on how the concept of hypothesis testing, null hypothesis, alternative hypothesis, rejection regions, etc. can be explained in a simple(!) and intuitive way - I would love to know how.
 A: Working with soda sounds fun, and the test of whether teenagers can actually tell the difference between sodas makes sense once you have a reasonable knowledge of hypothesis testing. The problem might be that this question: "can you actually tell the difference between sodas?" is complicated by lots of other stuff in the minds of teens, like "who is good and who is bad at testing sodas?", "is there actually any difference between the sodas?"
I've never taught teens stats, but I've always fantasized about using a loaded die, or biased coin. Die more interesting, but statistically more challenging. With the coin example, a coin either is or is not fair. There's no being good at flipping coins. There's no deciding whether it's heads or tails. 
If we flip a coin for who wins $100, and it comes up heads (you win!), I might say, "Hey. How do I know whether that coin is fair? I bet you rigged the competition!". You say "Oh yeah? Prove it." The fairly obvious solution is to flip the coin over and over to see whether it comes up more heads than tails. We flip it, and it comes up heads. "Ahha! I say. Seee! It's biased towards heads!" And so on. 
Good biased coins don't exist, but biased dice do -- you can buy one on Amazon. You could offer students a prize if they can win some number of rolls. But you know you'll win. They'll be angry. You say, OK, I'll give you the prize if you can prove this die is biased, with say, 95% confidence.
Then move on to soda. The prize could even be a soda party! "Hey, I wonder whether you guys can tell the difference between coke and pepsi..."
A: Consider someone doing target practice with a shotgun, which shoots bursts of pellets in the direction of the barrel.
Null Hypothesis: I'm a good shooter, and my barrel is perfectly on target. Not left, not right, but straight at it. My error is 0.
Alternate Hypothesis: I'm a bad shooter, and my barrel is off target. Just left or just right of the target. My error is e>0 or e<0.
Since any measurement has a certain average error (i.e. standard error), a measurement that says "off target" is possible, even if I'm shooting straight. I'll need to not "hit" my target (at all, even with each shot being a burst/spread) a certain number of times, before you can call me a bad shooter and choose the Alternate Hypothesis.
A: I think you should start with asking them what they think it really means to say about a person that he or she is able to tell the difference between coca-cola and pepsi. What can such a person do that others can not do?
Most of them will not have any such definition, and will not be able to produce one if asked. However, a meaning of that phrase is what statistics gives us, and that is what you can bring with your "a taste for statistics" class.
One of the points of statistics is to give an exact answer to the question: "what does it mean to say of someone that he or she is able to tell the difference between coca-cola and pepsi"
The answer is: he or she is better than a guessing-machine to classify cups in a blind test. The guessing machine can not tell the difference, it simply guesses all the time. The guessing machine is a useful invention for us because we know that it does not have the ability. The results of the guessing machine are useful because they show what we should expect from someone who lacks the ability that we test for.
To test whether a person is able to tell the difference between coca-cola and pepsi, one must compare his or hers classifications of cups in a blind test to the classification that a guessing machine would do. Only if s/he is better than the guessing machine, s/he is able to tell the difference.
How, then, do you determine whether one result is better than another result? What if they are almost the same?
If two persons classify a small number of cups, it's not really fair to say that one is better than the other if the results are almost the same. Perhaps the winner just happened to be lucky today, and the results would have been reversed if the competition was repeated tomorrow?
If we are to have a trustworthy result, it can not be based on a tiny number of classifications, because then chance can decide the result. Remember, you don't have to be perfect to have the ability, you just have to be better than the guessing machine. In fact, if the number of classifications is too small, not even a person that always identifies coca-cola correctly will be able to show that s/he is better than the guessing machine. For example, if there is only one cup to classify, even the guessing machine will have 50 per cent chance to classify completely correct. That's not good, because that means that in 50 per cent of the trials, we would falsely conclude that a good coca-cola identifier is no better than the guessing machine. Very unfair.
The more cups there are to classify, the more opportunities for the guessing machine's inability to be revealed and the more opportunities for the good coca-cola identifier to show off.
10 cups might be a good place to start. How many right answers must a human then have to show that he or she is better than the machine?
Ask them what they would guess.
Then let them use the machine and find out how good it is, i.e. let all pupils generate a series of ten guesses, eg. using a dice or a random generator on the smartphone. To be pedagogical, you should prepare a series of ten right answers, which the guesses are to be evaluated against.
Record all the results on the board. Print the sorted results on the board. Explain that a human would have to be better than 95 per cent of those results before a statistician would acknowledge his or her ability to tell the difference between coca-cola and pepsi. Draw the line that separates the 95% worst results from the top 5% results.
Then, let a few pupils try classifying 10 cups. By now the pupils should know how many right they need to have to prove that they can tell the difference.
All this is not really doable in 10 minutes though.
A: Assume the kids can't tell the difference and decide by chance. Then each kid has a 50% chance of guessing it right. So you expect (expected value) that in this case, 5 kids do it right and 5 kids err. Of course, as it is by chance, it is also possible that 6 kids err and 4 get it right, and so on. On the opposite side, even if the kids can tell the difference, it is possible, that by chance one of them errs.
Intuitively, it is clear, that if the kids guess by chance, it is rather improbable that all kids give the correct answer. In this case one would rather believe that the kids actually could taste the difference between both drinks. In other words, we don't expect improbable events to be observed. So if we observed an event that is improbable under the 50-50 scanario, we rather believe that this scenario is false and the kids can distinguish between Coke and Pepsi.
But what does "rather improbable" and "rather believe" mean? Let your pupils choose $\alpha$: "If we observe an event from the extreme end contradicting the 50-50 hypothesis, which probability may it at most have such that you don't believe this hypothesis any more?" Hope they don't answer $\alpha \leq 0.00098$ Write their $\alpha$ at the board. I assume $\alpha=0.05$. So you and your pupils agree: If we observe an event that belongs to the upper 5% of extreme events contradicting the 50-50 scenario, we don't belive in this scenario any more (reject the hypothesis). 
Now calculate the binomial distribution with them. $P(\textrm{all kids guess it right})=0.00098$, $P(\textrm{only one kid confuses Coke with Pepsi})=0.01074$ 
 and $P(\textrm{only two kids confuse})=0.05468$. Obviously, you'll only conclude that there is a difference between both beverages, if at most one kid confuses them.
This is the moment where you conduct the experiment. Do it thoroughly with all 10 pupils, even if you just calculated that you could stop after the second error. Then record the results and keep them. You'll need the results if you want to explain meta-analyses to them. 
(By the way, the historical example is about tasting if the milk or the tea has been poured first into the cup. The tea tasting lady.)
A: Show this video which is the most intuitive explanation of hypothesis testing that I have ever seen - https://www.youtube.com/watch?v=UApFKiK4Hi8
A: The children tasting coke experiment is a good example to introduce hypothesis testing, as its equivalent the lady tasting tea experiment showed. However, evaluating those experiments is not very intuitive because the null hypothesis involves the binomial distribution with p=0.5, and it is not straightforward.
In my usual introduction to hypothesis testing, I try to overcome this drawback by using only the all-successes case in the binomial distribution, whose probability can be computed as p^n even by people who doesn't know about the binomial probability.
In my favourite example, I like roasted chestnuts and I buy a handful of them from a street vendor. I get them at a discount price because they come from a big bag where 10% of chestnuts have a worm hole - here I try to make clear that the bag has been well mixed so that my handful of chestnuts is a random sample of the chestnuts in the bag and the vendor's statement means that every chestnut has an independent probability of 10% of having a worm hole.
As I start enjoying my roasted chestnuts, I take them one by one and check them for worm holes before eating them.
When I check the first chestnut, I see a worm hole, and I wonder if the vendor lied to me - I explain here that wondering that is setting my null hypothesis p=10% and my alternative hypothesis p>10%, and I put them in the blackboard. Do I have a reason to doubt that p=10% when I got one bad chestnut out of one? Well, 10% of people performing the same experiment would get the same result, so I can think I just had bad luck.
Then, I take the second chestnut and it has a worm hole, too. Two out of two has a probability of just 1% if the vendor hasn't lied to me. I could have had a very bad luck, but I get very suspicious about the vendor.
The third chestnut has a worm hole, too. Getting the three chestnuts with worms out of three wouldn't be impossible assuming that the vendor is fair and p=10%, but it would be very unlikely (probability=0.1%). Therefore now I have a strong reason to doubt on the vendor's work and I raise a complaint and ask to be refunded.
Of course, this kind of successive test have some theoretical problems, but it doesn't matter much to show the idea of an hypothesis tests. In fact, the most important idea that is not covered in that example is that in hypothesis tests we compute the probability of the results we get or anything worse - in my example this was avoided by just getting the worst possible result.
I have used this example several times with freshmen at university - which are still technically teenagers - but I think it could work well with younger teenagers, too.
