Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

Consider me a "naive researcher."

We have a test of 25 items grouped into 5 item types. Each item is correct or incorrect. All items are math questions, but we expected 4 item types to have different language confounds. We have a population of 28 group A and 32 group B. We expected group A to under perform group B in the 4 item types with language confounds.

Overall the test scores are negatively skewed but the SDs are similar between groups. In t-tests for each section we found a significant difference in those 4 item types in the expected direction and no significant difference for the fifth type. Cohen's D shows medium effect sizes in the sections with significant differences.

Does it make sense to estimate and report reliability for each of the item types? If so what would be the best estimate?

share|improve this question
Are the sections supposed to measure different constructs (e.g., linear algebra, real analysis, etc.) or do you consider the total responses correct on all 25 items? If sections correspond to different scales, do the four suspicious items belong to the same section or not? Finally, could you clarify what you mean by "language confound" and "reliability". I interpret the former as a possible effect of differential item functioning, but for reliability I suspect you mix it up with variability (or standard deviation). – chl♦ Jan 7 '12 at 11:24
Thank you for having a look. Group A is non-native speakers and group B is native speakers. One section of the test uses no language, just math (numbers and symbols). The other sections have various language issues. One for example has negative wording on all the questions (e.g., Which element rate did not fall?). So, although the test is supposed to be a math test, we assumed that these sections might at the same time be testing language, so that the two constructs are confounded. As for reliability, I'm thinking of measures such as Chronbach's alpha. I hope that clarifies. – Brett Reynolds Jan 7 '12 at 14:25
Ok, so my remark on possible DIF is meaningless. I would expect that language difficulty will decrease reliability or item-score point-biserial correlation in one of the two groups. Is that what you observed? (Sidenote: (1) If raw scores distributions are highly skewed, interpreting SDs alone does not make much sense; (2) negative item formulation associated to negative wording of proposed answers have been shown to yield poorer performance in general.) – chl♦ Jan 7 '12 at 20:24
I calculated point-biserial correlations for the whole population, but had not thought to do so for each group. I'll try it and get back to you, tomorrow. Thanks again. – Brett Reynolds Jan 8 '12 at 1:49

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.