Consider me a "naive researcher."
We have a test of 25 items grouped into 5 item types. Each item is correct or incorrect. All items are math questions, but we expected 4 item types to have different language confounds. We have a population of 28 group A and 32 group B. We expected group A to under perform group B in the 4 item types with language confounds.
Overall the test scores are negatively skewed but the SDs are similar between groups. In t-tests for each section we found a significant difference in those 4 item types in the expected direction and no significant difference for the fifth type. Cohen's D shows medium effect sizes in the sections with significant differences.
Does it make sense to estimate and report reliability for each of the item types? If so what would be the best estimate?