I have some categorical data and measures of participants accuracy. Let's say that it is a quiz and we have 8 different categories: History, Geography, Physics and so forth. Each participant is quizzed 25 times on each category and my first task was to figure out how accurate participants are expected to be in each category and determining whether there is a reliable difference in accuracy between the categories. Having established that my next goal is to investigate segmentation. Maybe rather than think of 8 separate categories, accuracy in physics and math could for instance be related or art or history. There are several theories out there about which categories (if any) should be related, so I am trying a data driven investigation.
As a sanity test, I am analyzing some simulated data and the full script can be found here. For each simulation the BRMS script seems to estimate the true values relatively well:
brm(Cor ~ Category + (1|Subject), family=bernoulli(link = "logit"), data = results)
But when I segment using tree:
tree(Cor~ Category, data=meanpercat)
Most of the time I get 3-4 segments but sometimes 2 or 5. In other words my current method for segmentation seems unreliable.
I see 2 potential benefits of the BRMS approach vs. the tree approach: 1) BRMS has a random intercept for participants and takes into account who are better and worse than average. 2) BRMS takes every single trial into account wheres tree only takes a mean for each participant for each category.
Are there any ways to get these benefits for my current approach or should I switch to another method? I haven't had a lot of experience with segmentation so I could easily be messing something.
I would really appreciate your feedback here