I have two groups of students, one group who received tutoring, the other who did not. I have the test scores for both groups in the beginning of the year (Mark1). I also have their test scores at the end of the year (Mark3). I would like to determine if the students who received tutoring improved more (or declined less) than students who did not receive tutoring.
Initially, I took the average of Mark3 and subtracted the average of Mark1 for each group.
| IsTutored | Students | Mark1 | Mark3 | Diff_1_3 |
|-----------|----------|-----------|-----------|-----------|
| FALSE | 68 | 60.863636 | 54.986842 | -3.573529 |
| TRUE | 31 | 58.6875 | 68.69697 | 9.516129 |
With the individual scores, I was able to get the confidence intervals for Diff_1_3 to find statistical significance.
The trouble is, tutored students and non tutored students did not start with the same mark. I'm trying to show improvement or weakening relative to where they started from. So I took the percentage change of the marks for each individual student, and then averaged the percentage change (Diff_1_3Pct:
| IsTutored | Students | Mark1 | Mark3 | Diff_1_3 | Diff_1_3Pct |
|-----------|----------|-----------|-----------|-----------|-------------|
| FALSE | 68 | 60.863636 | 54.986842 | -3.573529 | -0.040333 |
| TRUE | 31 | 58.6875 | 68.69697 | 9.516129 | 0.258208 |
I'm wondering about a few things:
- Is percentage change a good way to approach this?
- Because I take the percentage change of each student's mark and then average these to arrive at Diff_1_3Pct, Diff_1_3Pct is not the same as the percentage change of the average of Mark1 and the average of Mark3. For example, non tutored students dropped about 6 points from 60.8 to 54.9. This is a drop of about 10%. But Diff_1_3Pct is about -4%. Is this Simpson's Paradox?
- I could instead take the percentage difference of the average of Mark3 and the average of Mark1 (instead of the percentage difference of the individual scores), however if I did that, I'm not sure how I could calculate the confidence intervals
-- UPDATE -- At the suggestion of user158565, I ran a mixed linear model.
data.head()
+-----------+------+-----------+------+
| StudentID | test | isTutored | Mark |
+-----------+------+-----------+------+
| 2217214.0 | 0 | 1 | 81.0 |
+-----------+------+-----------+------+
| 2206910.0 | 0 | 0 | 51.0 |
+-----------+------+-----------+------+
| 2212272.0 | 0 | 1 | 44.0 |
+-----------+------+-----------+------+
| 2207991.0 | 0 | 0 | 72.0 |
+-----------+------+-----------+------+
| 2215861.0 | 0 | 1 | 50.0 |
+-----------+------+-----------+------+
Model:
import statsmodels.api as sm
import statsmodels.formula.api as smf
data = tutormarks_reshape[tutormarks_reshape.CourseNumber == '24414']
model = sm.MixedLM.from_formula("Mark ~ test*isTutored",data,groups=data["StudentID"])
result = model.fit()
print(result.summary())
Results:
Mixed Linear Model Regression Results
===========================================================
Model: MixedLM Dependent Variable: Mark
No. Observations: 198 Method: REML
No. Groups: 99 Scale: 198.8986
Min. group size: 2 Likelihood: -847.6510
Max. group size: 2 Converged: Yes
Mean group size: 2.0
-----------------------------------------------------------
Coef. Std.Err. z P>|z| [0.025 0.975]
-----------------------------------------------------------
Intercept 64.919 3.154 20.586 0.000 58.738 71.100
test -1.787 1.209 -1.477 0.140 -4.157 0.583
isTutored -10.064 5.636 -1.786 0.074 -21.110 0.981
test:isTutored 6.545 2.161 3.028 0.002 2.309 10.781
groups RE 179.006 4.009
===========================================================