How to create data agreeing with study results? I'm teaching hypothesis testing at a high school and came across this "study" on "Chocolate causes weight loss". I'd really like to look at it in class with my students and not only discuss its maths but also what the problems of this study (and many more, but probably less intentional) is.
Unfortunately, the study does not contain the raw data of the participants and since the participants are all identifiable now (appeared on a TV documentary), the author cannot make the data public.
I'd like to recreate more or less realistic data (just for the weight) for the three groups (low-carb, chocolate, control) such that it roughly matches the result given in the study.
The study only shows two figures picturing daily weight development by group and cumulated change of weight by group. It is known that there were 15 participants starting with an average weight of 81.5kg. The weight change after 21 days was documented to be -3.1% (low-carb), -3.2% (chocolate) and +0.7% (control). Lastly, they state "the weight reduction of this (chocolate) group exceeded the results of the low-carb group by 10% after only three weeks (p = 0.04)".
Question: What would be the easiest way to recreate such data?
Ideally, this method also works for very different data, so I can reuse it to create further exercises in the future. In other topics of maths, it is rather easy to "start from the solution" and create an exercise, but in statistics, I usually end up just using some numbers, hoping for the best and correcting until it works.
 A: Huy, I heartily encourage your interest in teaching statistics through simulation. The more I see of statistical [mal]practice in the social sciences (including medicine), the more convinced I become that simulation holds the key to a meaningful engagement with statistics at all levels.
For you and your students, I would discourage the use of any black-box tools. Rather, a spreadsheet that your students can inspect and modify themselves will give them the most proper exposure to these ideas. Also, to teach your students effectively, I think you will need to have built the spreadsheet yourself so that you understand in depth what's going on 'under the hood'.
To start, I would suggest you build a 5-column spreadsheet. The first 3 columns would give the potential outcomes for the study participants, one row for each subject. Column 1 will have the weight change that person would (or did) experience if put in the control group. (You could fill this column in with one of the random-number functions from your spreadsheet program.) The next 2 columns would have the weight changes that would happen in each of the 2 treatment groups. (These columns might be filled in by adding random 'treatment effects' to the control-group weight changes.) Column 4 would have a randomly-generated indicator telling which group the person actually got assigned to. Column 5 would use an appropriate formula to copy over the actual weight change from one of the columns 1..3; some if-then logic should suffice. Once you've done this, columns 4..5 will look like (very much simplified) raw data from the study.
You will have to decide how to generate the treatment effects, of course, and this work actually amounts to deciding what your model is! By suitable use of parameters entered into cells off to the side of the main $N \times 5$ grid of data, you can even explore how strong the 'Chocolate effect' would have to be to get detected in studies of different sizes. If you have 20 students in your classroom, you can even model a null effect, and show that 1 in 20 trials still (on average) gets a 'positive result'!
Once again, thank you for your work to bring these important matters alive in your classroom — during Math and Statistics Awareness Month, no less!
