2
$\begingroup$

I have collected questions from 10th grade students about genetic engineering before and after learning about the subject and classified the questions based on their cognitive level..

Before learning about the subject, I collected a total of 45 student questions. I classified 28 of those as "input" (basic) questions, 8 questions as "processing" (intermediate) and 9 as "output" (advanced).

After learning about the subject, I collected from the same students 28 questions, distributed as follows: 11 "input", 7 "processing" and 10 "output".

What test is appropriate to see if the distribution of student questions among the three categories is different? I tried conducting a prop.test but that only distinguishes between two categories (success and failure). Thanks.

Improve this question
$\endgroup$
2
  • $\begingroup$ Could you please explain what you mean by "distribution of student questions"? It appears that students provide answers, not questions and that the distributions of question types themselves have changed from pre-test to post-test. How exactly do you propose to go about incorporating the three categories of questions in measuring student progress? $\endgroup$
    – whuber ♦
    Jul 3, 2012 at 15:14
  • 1
    $\begingroup$ I asked the students to pose questions. Question-posing by students has come to be recognized as an important skill in science education. As you wrote, I am interested to see if the distribution of question types has changed from pre-test to post-test. As the students ask a larger proportion of advanced questions, we believe their comprehension of the material taught has grown fuller and deeper. (A full discussion is beyond the scope of this comment.) $\endgroup$
    – Aviv
    Jul 3, 2012 at 19:55

2 Answers 2

Reset to default
3
$\begingroup$

To simply test for differences in the proportions, a chi-squared test should do the trick. As your categories seem ordered, a CATT could have higher sensitivity.

In R:

z = matrix(c(28,8,9,11,7,10), nrow=2, byrow=T)

chisq.test(z)

prop.trend.test(z[1,],apply(z,2,sum))
Improve this answer
$\endgroup$
6
  • $\begingroup$ Given that the distribution of types of questions changed from pretest to post test, how do you propose to interpret the results of a chi-squared test? And how does this test account for the ordinal nature of the question types? (AFAIK, it does not.) $\endgroup$
    – whuber ♦
    Jul 3, 2012 at 13:42
  • $\begingroup$ As I understood the original post, the question is whether there is a difference in the level of sophistication of the students' questions between before and after learning. The chi-squared or fisher's exact test would do the job, and the cochrane-armitage trend test would perhaps do it even better. I don't quite understand the reason for your concern. What would you recommend? $\endgroup$
    – miura
    Jul 3, 2012 at 18:37
  • $\begingroup$ (+1) The C-A trend test looks like a better choice here. It is implemented as coin::independence_test. $\endgroup$
    – whuber ♦
    Jul 3, 2012 at 19:03
  • $\begingroup$ miura's interpretation is as I intended. I successfully used prop.trend.test. Will coin::independence_test yield different results? $\endgroup$
    – Aviv
    Jul 3, 2012 at 19:58
  • 1
    $\begingroup$ As mentioned here (stat.ethz.ch/pipermail/r-help/2008-December/182575.html), maybe marginally different. $\endgroup$
    – miura
    Jul 3, 2012 at 20:15
1
$\begingroup$

In small samples the $\chi^2$ test may not be good because the $\chi^2$ is an asymptotic distribution for the test statistic. Exact tests for contingency tables are always available. You can use the Fisher exact test (or generalization to $R \times C$ tables).

Improve this answer
$\endgroup$
7
  • $\begingroup$ I would direct your attention, if you please, to the comment I wrote to @miura's answer. To be perfectly clear: what is the basis for recommending a chi-squared test (or any variant) for this problem and how could the results be interpreted? $\endgroup$
    – whuber ♦
    Jul 3, 2012 at 15:13
  • $\begingroup$ @whuber Aviv has categorized the type of questions asked of the students not the answers. The question he poses is whether the types of questions asked after learning have the "same" distrbution as the types before learning> This is a simple comparison of two frrequency distributions to see if they are nearly the same. The chi square or the exact test for the contingency table is exactly the required test for this. $\endgroup$
    – Michael R. Chernick
    Jul 3, 2012 at 15:44
  • $\begingroup$ I wonder about that interpretation, Michael, because the OP refers to "success" and "failure." I suspect he wants to gauge student improvement and perhaps wishes either to break it down by type of question or somehow to compare the distributions of correct answers among the three categories of questions. This could (sort of) be done with a three-way chi-squared test but that would ignore the ordering of the question difficulty. $\endgroup$
    – whuber ♦
    Jul 3, 2012 at 16:46
  • $\begingroup$ @whuber I reread the question. The OP seems to be saying that the students are asking questions about genetic engineering before they have studied the subject and then they learn a little about the subject and ask additional questions after that adn the issue is whether or not the type of questions they ask is affected by teh knowledge they gained on the subject. The fact that there is an ordering of the level of the question really doesn't affect the problem because the contingency table checks by comparing the bins in one group to their counterpart in the other. It does seem complicated. $\endgroup$
    – Michael R. Chernick
    Jul 3, 2012 at 16:57
  • $\begingroup$ Ah! Thank you! I misunderstood the question. I agree now with just about everything you have said, with one minor qualification: because this is a small-sample situation, we probably want the most powerful tests we can find, and I can't help thinking that a test which "respects" the ordering of the "cognitive difficulty" variable ought to be more powerful for the kind of change the OP wants to measure than a chi-squared test would be. $\endgroup$
    – whuber ♦
    Jul 3, 2012 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.