0
$\begingroup$

In a school I work a group of teachers made the claim that by using a computerized tool for helping students to learn math will make them pass the course easily. I hypothesize that tool will make the students dependent and that will increase the number of failed students at the end of the course. We have two classrooms A and B. Classroom A was using the computerized took and classroom B was given the traditional method of following the lecture notes, this decision was made because the number of licenses that were limited. There was not specific criteria to assign one student to classroom A or B.

Each student was given two tests, apart of one mid-term exam and one final exam. I hypothesize that students from the A classroom, the one that was using the software tool, will have a rate of failed students more than 60% (only 25% of the students passed the course). When I made a hypothesis testing I obtained a result of a p-value of 0.0156 so my alternative hypothesis hold (more than 60% will fail the course).

In the B classroom, where the students did not use any computerized took, the students who passed the course were approximately 69%.

The question that I have is how can I support my claim that the use of this software tool is not good at all for the students? I mean, how can I know that this was not only caused by random variation?

Thanks

$\endgroup$
  • $\begingroup$ casualty (at least in a particular sense of the word) is possibly correct but not really idiomatic for this context. If you intend to refer to "caused by non-random assignment to treatment" you should probably state that. If instead you mean "caused by random variation" you should probably state that. If you mean either or both it's probably better to state those explicitly as well (as the answers two the two possibilities are different). $\endgroup$ – Glen_b Oct 19 '16 at 23:48
  • $\begingroup$ (1) compute a statistical test on difference between the mean scores of classroom A and B. (2) The trickier issue is ascertaining what causes the effect: (i) do the different classrooms have different teachers? Might the teachers have significantly different skill? (ii) is there any scheduling reason why more mathematically talented kids would end up in class B rather than class A? (eg. maybe class A conflicts with another class the top students take)? Something to check is whether the treatment group (i.e. class A) and control group (i.e. class B) differ significantly on observable variables. $\endgroup$ – Matthew Gunn Oct 22 '16 at 2:59
  • 1
    $\begingroup$ My concern biggest concern would not be that the effect was caused by random variation. My biggest concern would be that better students systematically ended up in class B for some reason. You can strengthen your case (or find flaws in it) by systematically thinking of alternative stories to a bad software tool and see if you can rule them out. $\endgroup$ – Matthew Gunn Oct 22 '16 at 3:02
2
+25
$\begingroup$

This is really a question about research design and methodology, not statistics.

As with any type of claim, you have to marshal evidence and make a case by ruling out other potential explanations. Statistical analysis is one form of evidence you can bring to bear. But in the case you describe, you have a challenge. Lacking random assignment of students to conditions means that your two groups are not balanced on variables that matter for the outcome. It is very challenging to rule out all potential confounds that could bias the results of, e.g., a t-test. So you have to build a case for the conclusion by ruling out plausible alternative explanations. One thing that springs immediately to mind is that perhaps the students in the traditional classroom were simply more able students to start, and thus more likely to pass whatever the method of instruction. Maybe students who are more likely to fail are also more willing to take a class with unorthodox approaches to instruction. Both of these are plausible alternative explanations that undermine your main claim. So you need to rule them out. Ruling out all plausible explanations (not all possible explanations, just the those that are reasonable) is necessary in your case.

Putting that aside for the moment, the hypothesis testing you did is irrelevant. The question of interest is: did more students in class A pass a course than students in class B? The comparison to an arbitrary benchmark (e.g. 60%) not necessary for that question. A simple test of proportions is more appropriate...except, you don't have random assignment and so need to account for confounding variables.

However, from the sound of your question, you are not conducting a research project or writing up an academic article. It sounds like you are trying to convince your colleagues that something is not working for the students. (Is this right or am I totally off base?) So the standards are a bit different. You have some statistical evidence and then you have anecdotal and contextual evidence. Were I in that position, I might conduct and simple test of proportions and then appeal to the common knowledge of the school, teachers, and students. Maybe the teachers have a sense that the students in both classes were equally capable. Maybe they know that the teachers were both equally good. They know that the students and teachers in both classrooms were subject to the same policies and shocks that could impact student performance. In such a case, the test of proportions is more compelling. Heck, even just an eyeball look at the different rate of failure is sufficient. We only need statistics if we are making inferences about a population.

So, how to approach this really depends on your context, your audience, and what you want to achieve. In any case, you need to make a claim, present evidence, and rule out alternative explanations. You won't get causality, but you can make a strong case.

$\endgroup$
  • $\begingroup$ I was trying to write an academic article about the findings. About how the students were assigned to each classroom was at random and understanding level of the subject was the same for both groups at the beginning. $\endgroup$ – Layla Oct 22 '16 at 8:51
  • 1
    $\begingroup$ @Layla. I see. Ok, so ignore that last paragraph! If you have random assignment, you are in a much better position. In terms of statistical tests, you need to clarify your hypothesis. It sounds like you want to compare the outcomes between A and B. What is the outcome you care about--course failure or course grades? In the former case, your sample size is small enough that you could run a Fisher's Exact test. In the latter, a comparisons of means using a t-test would probably suffice. $\endgroup$ – paqmo Oct 22 '16 at 18:56
0
$\begingroup$

Agree with Paqmo's concerns about confounding variables.

Additionally, you ask: "The question that I have is how can I support my claim that the use of this software tool is not good at all for the students?" (emphasis mine). I don't think you can answer that with the current approach. Learning enough to pass a test is just one possible benefit. Others include learning rate, retention, motivational effects, and meta-cognitive impacts (e.g. learning how to learn). You would want to identify and measure the different plausible impacts you care about before concluding something as strong as "not good at all".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.