3
$\begingroup$

Suppose I administer an exam consisting of 320 multiple-choice questions (i.e., the answer is either correct or incorrect - there's no partial credit). I grade the exams, and now I want to do an analysis of the results where I would like to break the analysis up by how difficult the questions were.

I would like to talk about "the hardest 10% of questions" (i.e., the 32 questions for which the fewest numbers of students got the correct answer), as well as "the second hardest 10% of questions" (i.e., the 32 questions that were hardest for students, besides the ones that were in the hardest 10%), etc.

However, it is cumbersome to talk about "the fifth hardest 10% of questions" or "the eighth hardest 10% of questions". Even "the second hardest 10% of questions" sounds a bit silly.

It feels like there must be a descriptive statistic that gives a pithy name to these quantities, but I haven't been able to find one! Can you please give me some suggestions?

The first one that came to mind for me is percentile, but this isn't quite right, since "questions above the 90th percentile in difficulty" is already a bit long-winded, and describing the "second hardest 20% of questions" seems to require a description like "questions above the 80th percentile in difficult, but below the 90th percentile".

I could describe things in terms of rank, which results in pithy statements (good!) but isn't normalized to the number of questions:

"Question difficulty rank: 1st - 32nd" "Question difficulty rank: 33rd - 64th" "Question difficulty rank: 289th - 320th"

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
4
$\begingroup$

The main word is "quantile". A common example of this is "percentile".

For the specific case, "decile" (one-tenth of the distribution) or "quintile" (one-fifth of the distribtuion) might be appropriate. Another commonly used quantile is "quartile" (one-fourth).

You could then say something like "Decile 1 being the 10% most difficult questions and decile 10 being the 10% easiest questions ..." and then carry on with statements like "in decile 3 (description) ..."

E.g.:

decile 1: 20% below average overall grade; decile 2: 15% below average overall grade ... decile 10: 10% above average overall grade

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi nathanwww. Thanks for your suggestion. To make sure I understood correctly: are you suggesting I redefine decile? Because the "first decile" conventionally has the meaning that it is the value of the difficulty such that 90% of problems are easier than this value. Are you suggesting we should define "Decile 1" to mean the 10% hardest problems themselves? $\endgroup$
    – J.A
    Apr 9, 2018 at 5:37
  • $\begingroup$ Maybe decile 10 as hard and decile 1 as easy would be better. Imo, it doesn't matter as long as it's defined explicitly and applied consistently within a document. $\endgroup$
    – nathanwww
    Apr 9, 2018 at 5:42
  • $\begingroup$ The original meaning (which remains valid) of various quantiles or percentiles was indeed that of the values defined by a certain fraction (percent) of values being less and the complementary fraction (percent) being more, with small print for handling of ties. However, this meaning has been extended (positive view) or eroded (negative) to the bins, classes or intervals so defined, such as the first decile or quartile being also the interval below the first decile or quartile. After some years of trying to fight this, I have come to see that as futile, as such usage is entrenched. $\endgroup$
    – Nick Cox
    Apr 9, 2018 at 15:51
  • $\begingroup$ If you're uncomfortable with it, you need to find some equivalent wording such as the lowest tenth or quarter of the data. Turn and turn about, I guess that a good reason for the extended meaning is where you {OP] came in, namely that many of the alternative wordings seem clumsy and long-winded. $\endgroup$
    – Nick Cox
    Apr 9, 2018 at 15:52
0
$\begingroup$

Words such as "quartile", "quintile", "decile" can be used to refer to the values that divide ranked groups, or they can refer to the groups themselves. See Wiktionary entry for quartile.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.