1
$\begingroup$

I have some categorical data and measures of participants accuracy. Let's say that it is a quiz and we have 8 different categories: History, Geography, Physics and so forth. Each participant is quizzed 25 times on each category and my first task was to figure out how accurate participants are expected to be in each category and determining whether there is a reliable difference in accuracy between the categories. Having established that my next goal is to investigate segmentation. Maybe rather than think of 8 separate categories, accuracy in physics and math could for instance be related or art or history. There are several theories out there about which categories (if any) should be related, so I am trying a data driven investigation.

As a sanity test, I am analyzing some simulated data and the full script can be found here. For each simulation the BRMS script seems to estimate the true values relatively well:

brm(Cor ~ Category + (1|Subject), family=bernoulli(link = "logit"), data = results)

But when I segment using tree:

tree(Cor~ Category, data=meanpercat)

Most of the time I get 3-4 segments but sometimes 2 or 5. In other words my current method for segmentation seems unreliable.

I see 2 potential benefits of the BRMS approach vs. the tree approach: 1) BRMS has a random intercept for participants and takes into account who are better and worse than average. 2) BRMS takes every single trial into account wheres tree only takes a mean for each participant for each category.

Are there any ways to get these benefits for my current approach or should I switch to another method? I haven't had a lot of experience with segmentation so I could easily be messing something.

I would really appreciate your feedback here

$\endgroup$
1
$\begingroup$

So first off, I don't think using brms is particularly justified, especially since you are using default priors. You might as well just use glmer or similar methods if you are going to use default priors.

With that aside, I think a mixed effect model might be interesting. You mentioned that physics and math might be similar, which makes me wonder if you could project the data onto a lower dimensional space via PCA. Since every person is quizzed in every subject, you could create a Subject by Category design matrix, where each cell is the person's average score in that category, and then do PCA.

These are just thoughts. It would be better if we had actual data to work with.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your repiy. If I understand you correctly, we already have the matrix you request in the: 'meanpercat <- aggregate(Cor ~ Subject + Category, results, mean) ' PCA is also something I have considered, but my worry is that PCAs are more meant for prediction than for interpretation. It is ok to be able to predict a participants score, but it is more important to know WHY we make that prediction. I may misunderstand you though and in that case please refer me to resources or clarify. If you absolutely need my data, I can share it but it has the structure of the simulated. $\endgroup$ – Simon Hviid Del Pin Nov 1 '19 at 6:00
  • $\begingroup$ Since you only have a few categories, you can likely take a look at the loading factors and come to a conclusion about what the principal components represent. $\endgroup$ – Demetri Pananos Nov 1 '19 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.