how to support my claims in the following scenario? In a school I work a group of teachers made the claim that by using a computerized tool for helping students to learn math will make them pass the course easily. I hypothesize that tool will make the students dependent and that will increase the number of failed students at the end of the course. We have two classrooms A and B. Classroom A was using the computerized took and classroom B was given the traditional method of following the lecture notes, this decision was made because the number of licenses that were limited. There was not specific criteria to assign one student to classroom A or B.
Each student was given two tests, apart of one mid-term exam and one final exam. I hypothesize that students from the A classroom, the one that was using the software tool, will have a rate of failed students more than 60% (only 25% of the students passed the course). When I made a hypothesis testing I obtained a result of a p-value of 0.0156 so my alternative hypothesis hold (more than 60% will fail the course).
In the B classroom, where the students did not use any computerized took, the students who passed the course were approximately 69%.
The question that I have is how can I support my claim that the use of this software tool is not good at all for the students? I mean, how can I know that this was not only caused by random variation?
Thanks
 A: This is really a question about research design and methodology, not statistics.
As with any type of claim, you have to marshal evidence and make a case by ruling out other potential explanations. Statistical analysis is one form of evidence you can bring to bear. But in the case you describe, you have a challenge. Lacking random assignment of students to conditions means that your two groups are not balanced on variables that matter for the outcome. It is very challenging to rule out all potential confounds that could bias the results of, e.g., a t-test. So you have to build a case for the conclusion by ruling out plausible alternative explanations. One thing that springs immediately to mind is that perhaps the students in the traditional classroom were simply more able students to start, and thus more likely to pass whatever the method of instruction. Maybe students who are more likely to fail are also more willing to take a class with unorthodox approaches to instruction. Both of these are plausible alternative explanations that undermine your main claim. So you need to rule them out. Ruling out all plausible explanations (not all possible explanations, just the those that are reasonable) is necessary in your case. 
Putting that aside for the moment, the hypothesis testing you did is irrelevant. The question of interest is: did more students in class A pass a course than students in class B? The comparison to an arbitrary benchmark (e.g. 60%) not necessary for that question. A simple test of proportions is more appropriate...except, you don't have random assignment and so need to account for confounding variables. 
However, from the sound of your question, you are not conducting a research project or writing up an academic article. It sounds like you are trying to convince your colleagues that something is not working for the students. (Is this right or am I totally off base?) So the standards are a bit different. You have some statistical evidence and then you have anecdotal and contextual evidence. Were I in that position, I might conduct and simple test of proportions and then appeal to the common knowledge of the school, teachers, and students. Maybe the teachers have a sense that the students in both classes were equally capable. Maybe they know that the teachers were both equally good. They know that the students and teachers in both classrooms were subject to the same policies and shocks that could impact student performance. In such a case, the test of proportions is more compelling. Heck, even just an eyeball look at the different rate of failure is sufficient. We only need statistics if we are making inferences about a population. 
So, how to approach this really depends on your context, your audience, and what you want to achieve. In any case, you need to make a claim, present evidence, and rule out alternative explanations. You won't get causality, but you can make a strong case.
A: Agree with Paqmo's concerns about confounding variables.
Additionally, you ask: "The question that I have is how can I support my claim that the use of this software tool is not good at all for the students?" (emphasis mine). I don't think you can answer that with the current approach. Learning enough to pass a test is just one possible benefit. Others include learning rate, retention, motivational effects, and meta-cognitive impacts (e.g. learning how to learn). You would want to identify and measure the different plausible impacts you care about before concluding something as strong as "not good at all".
