Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

I have collected questions from 10th grade students about genetic engineering before and after learning about the subject and classified the questions based on their cognitive level..

Before learning about the subject, I collected a total of 45 student questions. I classified 28 of those as "input" (basic) questions, 8 questions as "processing" (intermediate) and 9 as "output" (advanced).

After learning about the subject, I collected from the same students 28 questions, distributed as follows: 11 "input", 7 "processing" and 10 "output".

What test is appropriate to see if the distribution of student questions among the three categories is different? I tried conducting a prop.test but that only distinguishes between two categories (success and failure). Thanks.

share|improve this question
Could you please explain what you mean by "distribution of student questions"? It appears that students provide answers, not questions and that the distributions of question types themselves have changed from pre-test to post-test. How exactly do you propose to go about incorporating the three categories of questions in measuring student progress? – whuber♦ Jul 3 '12 at 15:14
1  
I asked the students to pose questions. Question-posing by students has come to be recognized as an important skill in science education. As you wrote, I am interested to see if the distribution of question types has changed from pre-test to post-test. As the students ask a larger proportion of advanced questions, we believe their comprehension of the material taught has grown fuller and deeper. (A full discussion is beyond the scope of this comment.) – Aviv Jul 3 '12 at 19:55

2 Answers

up vote 3 down vote accepted

To simply test for differences in the proportions, a chi-squared test should do the trick. As your categories seem ordered, a CATT could have higher sensitivity.

In R:

z = matrix(c(28,8,9,11,7,10), nrow=2, byrow=T)

chisq.test(z)

prop.trend.test(z[1,],apply(z,2,sum))
share|improve this answer
Given that the distribution of types of questions changed from pretest to post test, how do you propose to interpret the results of a chi-squared test? And how does this test account for the ordinal nature of the question types? (AFAIK, it does not.) – whuber♦ Jul 3 '12 at 13:42
As I understood the original post, the question is whether there is a difference in the level of sophistication of the students' questions between before and after learning. The chi-squared or fisher's exact test would do the job, and the cochrane-armitage trend test would perhaps do it even better. I don't quite understand the reason for your concern. What would you recommend? – miura Jul 3 '12 at 18:37
(+1) The C-A trend test looks like a better choice here. It is implemented as coin::independence_test. – whuber♦ Jul 3 '12 at 19:03
miura's interpretation is as I intended. I successfully used prop.trend.test. Will coin::independence_test yield different results? – Aviv Jul 3 '12 at 19:58
1  
As mentioned here (stat.ethz.ch/pipermail/r-help/2008-December/182575.html), maybe marginally different. – miura Jul 3 '12 at 20:15
show 1 more comment
up vote 1 down vote

In small samples the $\chi^2$ test may not be good because the $\chi^2$ is an asymptotic distribution for the test statistic. Exact tests for contingency tables are always available. You can use the Fisher exact test (or generalization to $R \times C$ tables).

share|improve this answer
I would direct your attention, if you please, to the comment I wrote to @miura's answer. To be perfectly clear: what is the basis for recommending a chi-squared test (or any variant) for this problem and how could the results be interpreted? – whuber♦ Jul 3 '12 at 15:13
@whuber Aviv has categorized the type of questions asked of the students not the answers. The question he poses is whether the types of questions asked after learning have the "same" distrbution as the types before learning> This is a simple comparison of two frrequency distributions to see if they are nearly the same. The chi square or the exact test for the contingency table is exactly the required test for this. – Michael Chernick Jul 3 '12 at 15:44
I wonder about that interpretation, Michael, because the OP refers to "success" and "failure." I suspect he wants to gauge student improvement and perhaps wishes either to break it down by type of question or somehow to compare the distributions of correct answers among the three categories of questions. This could (sort of) be done with a three-way chi-squared test but that would ignore the ordering of the question difficulty. – whuber♦ Jul 3 '12 at 16:46
@whuber I reread the question. The OP seems to be saying that the students are asking questions about genetic engineering before they have studied the subject and then they learn a little about the subject and ask additional questions after that adn the issue is whether or not the type of questions they ask is affected by teh knowledge they gained on the subject. The fact that there is an ordering of the level of the question really doesn't affect the problem because the contingency table checks by comparing the bins in one group to their counterpart in the other. It does seem complicated. – Michael Chernick Jul 3 '12 at 16:57
Ah! Thank you! I misunderstood the question. I agree now with just about everything you have said, with one minor qualification: because this is a small-sample situation, we probably want the most powerful tests we can find, and I can't help thinking that a test which "respects" the ordering of the "cognitive difficulty" variable ought to be more powerful for the kind of change the OP wants to measure than a chi-squared test would be. – whuber♦ Jul 3 '12 at 17:00
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.