I am running an experiment and now it is time to do some analysis, but I am having a hard time figuring out the right way to analyze my data. I have a number of questions, but first let me give you some context.
This is a behavioral experiment, where the setting is as follows:
i) There are 2 conditions, let's call them A and B.
ii) Each participant is randomly assigned to either A or B.
iii) Regardless of the condition, each participant goes through 8 trials.
iv) At each trial, we show a stimuli to the participant, and then record whether they exhibit a certain behavior. So we have a binary variable, call it Target_Behavior, which takes the value 1 is the participant exhibits this behavior, and 0 otherwise.
The first thing I want to study is whether there is a significant difference between the two conditions, regarding the frequency participants exhibit the target behavior. As you can see, for each participant we have 8 measurements, so it is a repeated measures setting, but I don't know how to choose the right test that takes this into account.
Furthermore, I don't know whether my data allows for any parametric assumptions, so I am only using non-parametric tests, which is detrimental to the power of my tests.
So far, I have been using a very simple (and probably wrong) solution. I am looking at each trial on its own. This way, for any trial we have only a single measurement for each participant, so, for example considering trial 1, I make a list containing the values of Target_Behavior for condition A, and the same thing for condition B. Finally, I do a chi-square test, comparing the two lists.
I am pretty sure this is suboptimal though, while it also poses an extra challenge. I am convinced there is a significant difference between conditions A and B, but the test I just described only gives significant results if there is a difference of about 30% in frequency between the two conditions. If they are at 20% it turns out to be not significant, however, this doesn't seem right to me, 20% is a huge discrepancy!
I have about 20 participants in each condition, if this is of any help.
I would really appreciate your help in figuring the right way to deal with this situation. I have been looking online for weeks, and still I haven't found a solution. I have some additional questions, but for now I think this is the most important one.
Thank you in advance!