I am trying to figure out the best way to design this experiment:

Each day for a month, a group of people is given a survey containing two questions: 1) What is your overall mood today? [6 point scale 0-5] and 2) Did you meet with your supervisor today? [yes or no]. I want to measure the difference in mood caused by meeting with a supervisor and provide a 95% CI.

Everyone is encouraged to participate everyday, however participation is voluntary. From a similar study, we expect around 1/3 of people to have a participation rate of 80-100%, 1/3 to have 50-80%, and 1/3 to be below 50%.

I was thinking about using a paired test, where for each participant I look at the average mood score when they met with a supervisor compared to when they didn't. One of the issues with this is there might be people that don't have both positive and negative responses to the second question. Also, there could be different numbers of responses per participant (e.g. 10 negative vs. 2 affirmative). I'm not sure how much of an issue this is and if it needs to be (or can be) controlled for.

Alternatively, I could do two sample t test where I separate Q2 affirmatives from negatives and average per person. I'm a little unsure the best way to go about this. If we think of the "no" group as the control and the "yes" group as the treatment, it seems wrong to have people show up in both groups. Should I randomly assign participants to one or the other? What if I assign someone to a group, say the affirmative, but they have no affirmative responses? And would this method be preferable to the paired scenario?

  • $\begingroup$ Survey are more complicated than random experiments because you cannot typically use formulas you find in most elementary textbooks. In order to analyze the data properly, you'll need to take into consideration the way the survey is designed, how individuals were sampled to be included into the survey, finite population corrections, etc. Have you considered these complicating details? Your proposed survey is complicated by the fact that you intend to carry out repeated measurements as a panel survey. $\endgroup$ – StatsStudent Feb 12 at 4:34
  • $\begingroup$ It also appears that you will have to handle the problem of self-selection into treatments. The people who see their supervisor frequently may be inherently different from the people who do not and your analysis may be measuring this effect rather than the effect of the supervisor. For example, perhaps people who meet with their managers more often are they themselves managers, and managers generally have a better mood because they are paid better and less stressed about money. Unless taken into account, you'd be measuring this effect AND manager meetings, not just the meeting effect. $\endgroup$ – StatsStudent Feb 12 at 4:39
  • $\begingroup$ So, in short, given the complexity of your survey, you should think about meeting a statistical consultant for some guidance and planning. $\endgroup$ – StatsStudent Feb 12 at 4:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.