Experiment design: Researchers ran an experiment to test whether paying people to get up on time made them less likely to sleep in late. Fifty participants were randomized to 2 groups: a control group who got no incentive, or a group who got a small daily payment. The main outcome measure was the average time participants got out of bed after 16 weeks. Results: Compared with the control group, the daily payments group got up a mean of 9.2 minutes earlier (t = 1.21; 95% CI, -3.20 to 12.66; P = .23) than in the no-incentives group (4.4 minutes).

  • $\begingroup$ It's possible that I misunderstood your confidence interval. Is it (-3.2, 12.66) around 9.2? Or is it (-3.2, 12.66) around 9.2-4.4 = 4.8? $\endgroup$ – R Carnell Mar 22 at 19:02

I would recommend not paying, of course. I would also recommend changing the way that the results are reported. "There was insufficient evidence to conclude that an incentive of X was successful at reducing the average time participants got out of bed."

Lastly, if the researcher wants to continue down this line of research, there are a couple of ways to change the experiment:

  1. Create an experimental design with varying incentive. All that was really proved is that the particular incentive wasn't successful. Trying larger incentives and analyzing the results with a regression might suggest the incentive necessary for a statistically significant result.
  2. Try a different type of incentive all together. e.g. cash instead of coupons.
  3. Try a different end point. If wake-up time is too variable, maybe try bed time.
  4. I don't recommend this, but some researchers will just try the experiment again with a larger sample size under the assumption that the difference really was there, but they didn't have enough sample. I think it is a much better path to success to vary the incentive than just to increase the sample size to find again that there was no significant effect.
  • $\begingroup$ Shouldn't the mean be in the middle of the confidence interval? And if the no incentive group got up earlier maybe a non financial strategy could work? $\endgroup$ – JJEscp Mar 22 at 2:02
  • $\begingroup$ Yes. The mean is generally between the upper and lower endpoints of the confidence interval, but it doesn't have to be in the exact middle. If the no-incentive group got up earlier, it could be for a variety of reasons. (1) did they know they were in an experiment? That can change behavior. (2) was this experiment performed during the start of the pandemic? people might have changed their sleep behavior to adapt to working from home. (3) was the experiment performed during a change in the seasons r day light savings time? That can also affect sleep behavior. $\endgroup$ – R Carnell Mar 22 at 15:37
  • $\begingroup$ Thank you very much for your helpful comments R Carnell ! $\endgroup$ – JJEscp Mar 23 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.