2
$\begingroup$

I'm about to analyse some data (hypothesis testing) and would like some feedback on my approach as I have never seen this situation (missing cells in an ANOVA-like table). I would also like to know if a multi-level approach would be better (I have no experience with it, just curious to learn more). The data have the following repeated measures design:

    A1       A2
B1  A1B1    A2B1
B2  A1B2     .
B3  A1B3     .

(Sorry about formatting). As you can see, A2B2 and A2B3 do not exist (were not tested).

The first hypothesis is that y(A1B3) > y(A1B2) > y(A1B1), while the second is that y(A2B1) > y(A1B1). I would also like to compare y(A2B1) with y(A1B2) & y(A1B3), though these will be exploratory. The main reason for including this A2B1 was to verify that something did not go wrong as there is a well established effect (hence the hypothesis y(A2B1) > y(A1B1)). So I think that this is rather a separate question and, once verified, does not require further consideration (e.g. comparison with the other factors). That is the reason for the missing cells.

I am thinking about running a one-way ANOVA (effect of B holding A1 constant) for hypothesis 1 and a t-test (effect of A holding B1 constant) for hypothesis 2. As for the exploratory tests I was going to run two further t-tests. I think this is all ok but would like some feedback on it in case there is a problem with it.

As for the multi-level approach, the study involved a sample of participants all carrying out the same task. So it is hierarchical because there is a "single subject" level (each with ~30 trials in each condition/level), and then a "group" level on which I was planning to test the hypotheses on just the aggregated (means) single-subject data. So I wonder if multi-level is the way to go instead of ANOVA + t-tests?

Edit for clarification: there are 48 participants. By group level I just meant taking each subject's summary statistics and analysing those as in a typical ANOVA. As for the design, it is repeated measures and all subjects carry out the same task with the same conditions. Each trial is defined by belonging to one of the cells in the table.

Edit 2 for an example: Each trial of the experiment began with information about the upcoming target stimulus location, and this was either correct (A1) or incorrect (A2)*. When the target stimulus appeared in either the correct/incorrect location, it was either presented on its own (B1), with a distractor that was similar to the target (B2), or with one that was highly dissimilar (B3). The thing that makes it a bit strange is that *when a distrctor was presented, the preceding information was always correct (A1), hence the missing cells. So the factor A1/A2 was only "relevant" when there was no distractor, otherwise it was always just A1.

Improve this question
$\endgroup$
5
  • $\begingroup$ How many participants do you have? Can you explain the design a bit more ? What is the "group" level ? Aggregating the data will lose information. $\endgroup$
    – Robert Long
    Commented Jul 30, 2020 at 3:35
  • $\begingroup$ Hi @RobertLong I have edited the post to answer your questions. I am not sure how else I can explain the design. $\endgroup$
    – fffrost
    Commented Jul 30, 2020 at 7:11
  • $\begingroup$ OK thanks. Just so I am clear, A and B are both factors with 2 and 3 levels respectively. The combinations A2B2 and A2B3 were not applied, and you are interested in a model that includes the interaction between A and B ? $\endgroup$
    – Robert Long
    Commented Jul 31, 2020 at 4:39
  • $\begingroup$ Yes exactly. Although I was under the impression an interaction wouldn't be possible since I can't compare across all levels of A with all those of B. But I am not sure, if it is possible to somehow do this that would be great. $\endgroup$
    – fffrost
    Commented Jul 31, 2020 at 6:22
  • $\begingroup$ @RobertLong just in case an example would be better than speaking abstractly then I have edited in a more detailed description of the setup $\endgroup$
    – fffrost
    Commented Jul 31, 2020 at 6:37

1 Answer 1

Reset to default
1
$\begingroup$

It might suffice to do a one-way ANOVA, examining all 4 groups not just the 3 you seem to have been considering. That uses all of the information you have to get a better pooled estimate of the within-cell variance. Test that the overall model is significant.

Then examine your pre-specified comparisons. You "verify that something did not go wrong" based on your comparison of A1B1 against A2B1. Then you can proceed with the other tests of interest among the levels of B within A1. Use appropriate correction for multiple comparisons.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.