4
$\begingroup$

I'm preparing my first ever experiment for my PhD and am currently facing some difficulties with statistics. Here is my draft experimental design:

  • 3 factors with 5, 3, 3 levels. As a result I have 45 conditions. (These numbers should be irrelevant for my question).
  • For each condition, I measure the time it takes a participant to answer a question. These questions are in a sense "generated" by the condition and are different for the different conditions.
  • This is a repeated-measures study, where all participants are measured in all conditions. So I get 45 data points per participant.

Here is the interesting bit: For each condition I can come up with multiple possible questions that test the response time in that condition. So I figured, why not ask each participant multiple questions per condition and record multiple response time values. More data points should be better after all. So I could for example get 90 data points per participant, 2 for each condition.

However, I am not sure how to handle this during the analysis:

  1. Is it a good idea to do these multiple measurements per condition and participant? Each question is rather short, so fatigue should not be an issue.
  2. How to best analyze such data? Should I just take the average of the multiple data values? Another option would be to pretend I have double the number of participants, but then a "pair" of participants is not independent.
  3. Would this have an effect on how many participants I need?

Any other advise on this issue is much appreciated.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
3
$\begingroup$

  1. Yes, more data as a general rule is usually better and given that you have just one answer per condition, then the move to get 2 data points per condition for each participant is a good idea. Its not just 1 more data point, but also double the amount of data on which to model a response per person.

  2. You could take the average of the participant's answers. A strict answer to that depends on the variability between the answers. BUT, why bother, just add another factor for each participant 'question number' with two levels '1' and '2'. If there is no difference between the answer order, then the whole model will be the same as taking the mean. On the other hand, if the there is a systematic difference between the answers (for some reason), you can find out about that as well, essentially for free.

  3. 'Need' is a tricky word here, but yes, this could potentially decrease the amount of participants you need. Asking more questions increases your N, having them be within subject comparisons is even better (usually). So by asking more questions per person you should be reducing your variance, and thus increasing the likelihood of finding a significant model.

Improve this answer
$\endgroup$
8
  • $\begingroup$ This is already helpful, thanks! I am in a similar situation where I have 8 measurements over the course of 50 minutes, within-subjects (so 16 total). Therefore I am wondering whether you could elaborate on "If there is no difference between the answer order, then the whole model will be the same as taking the mean."? $\endgroup$
    – Steven Jeuris
    Commented Jan 22, 2015 at 17:23
  • $\begingroup$ In addition, don't you run into the risk of interpreting significant effects incorrectly? If the hypothesis is to expect a significant difference in the mean values, and there are only a limited amount of measurements, ... what does a significant effect even say in this case? I thought hypotheses had to be phrased up front, and tested subsequently. $\endgroup$
    – Steven Jeuris
    Commented Jan 22, 2015 at 17:25
  • 1
    $\begingroup$ @StevenJeuris I saw your other comment and sort of understand what you mean. If you had two groups with 8 measures each over 50 minutes, if there is no effect of time then the main effect of 'group' is equivalent to taking the mean of the 8 time points for each person (go ahead and try it out by calculating the mean and re-running the same analysis). If however there is an effect of time, than your model is a better one than the mean model, and you give yourself the potential of seeing an interaction between the groups and time (e.g. group 1 only differs in measurement at the end of the test") $\endgroup$
    – Mensen
    Commented Jan 22, 2015 at 20:06
  • $\begingroup$ @Mensen Thanks a lot for your answer and clarifications. Do you have any advice or any pointers to a good read regarding my 3rd question - namely how many participants would I need in such a study? $\endgroup$
    – Dimitar Asenov
    Commented Jan 23, 2015 at 8:23
  • $\begingroup$ @Mitko, the calculation for the minimum amount of data points (in your case participants) you need in order to potentially find a significant effect is statistical power analysis. This calculation essentially depends on how large your expected affect could be, the expected variability of the data, and your own parameter setting for the tolerance to type 2 errors (usually arbitrarily set to 0.8). That's the best answer I can give without going into the details of your study. $\endgroup$
    – Mensen
    Commented Jan 23, 2015 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.