What type of data analysis should I make in case of (quasi-) experimental data?

Question

In my school, there is a trend of using computer software for teaching a first course in Chemistry for undergraduate students. I hypothesized that the use of this tool could make the students achieve better scores than without this help. So far what I have done is the following:

• I chose two groups, both of them were going to follow the same topics. Only in one of them the use of the computer software was commendatory.
• I make an entrance test about basic chemistry knowledge to both groups and approximately 100 percent of both classes failed that examination.
• I had compiled the results of the mid-term exam and the final exam of both groups and I have seen an increase in the final marks of the group that was using the computerized tool.

I have read that this seems like a quasi experiment. I have gathered some information about this techniques, but apart from the basic stuff I am a little bit lost of what statistical techniques should I perform to gather a conclusion that this software tool should be encouraged to be used.

I have performed a hypothesis test, but I believe this is not enough. Any advice? My statistics knowledge is very basic.

Answer 1

The central question is whether you assigned students completely at random to the experimental groups.

If yes you can use a standard test for independent samples to assess the significance of the group difference. This is probably the hypothesis test that you have already done.

If no, it is indeed a quasi experiment also called a non-randomized experiment or broken experiment.

There are many statistical techniques available to correct for selective assignment including propensity score techniques, regression technqiues, and combinations of both.

The central question you need to answer to use these techniques is whether you know what caused the assignment to groups. Were certain students assigned to the computerized group (e.g. older/younger, better/worse, male/female, richer/poorer, smaller/taller....)? And do you have data on the variables determining the assignment? Then you need to adjust for the imbalance in these covariates using one of the above mentioned techniques.

The easiest, albeit not 'best', adjustment method is Analysis of Covariance, a regression technique which comes with the assumptions of linearity and normality in small samples. You can imagine it like a t-test where in addition you control for the confounding variables you identified as the responsible one for the treatment group assignment. Standard software packages like Excel, Stata, Spss, SAS or R have easy to use implementations of this technique.

Since you say that your statistics knowledge is very basic collaborating with a skilled analyst may be a good additional option in this situation, because the less basic techniques involving propensity scores can get a bit complicated.

Answer 2

please try propensity score matching ..here's a simple tutorial for you http://web.hku.hk/~bcowling/examples/propensity.htm ideally you divvy up the groups (like you have already tried) and ID different attributes of the students in both the groups , perform a logistic regression on both groups. For e.g. if you ID attributes like "score in maths" A and "score in science" B and score in chem is C then your equation would be something like C = xA+yB, which after logistic regression on different samples should give you coeffecient values of x and y .. post that you apply these very coeffecients to both the data sets and compare the mean/median ..what this would tell us is that due to A and B scores in C went up or down and that there's no bias in the test since all students are equally likely to be administered the computerized chem course .. if this is confusing, please read the tutorial above ..it does a better job