1
$\begingroup$

I have two groups of students, one group who received tutoring, the other who did not. I have the test scores for both groups in the beginning of the year (Mark1). I also have their test scores at the end of the year (Mark3). I would like to determine if the students who received tutoring improved more (or declined less) than students who did not receive tutoring.

Initially, I took the average of Mark3 and subtracted the average of Mark1 for each group.

| IsTutored | Students | Mark1     | Mark3     | Diff_1_3  |
|-----------|----------|-----------|-----------|-----------|
| FALSE     | 68       | 60.863636 | 54.986842 | -3.573529 |
| TRUE      | 31       | 58.6875   | 68.69697  | 9.516129  | 

With the individual scores, I was able to get the confidence intervals for Diff_1_3 to find statistical significance.

The trouble is, tutored students and non tutored students did not start with the same mark. I'm trying to show improvement or weakening relative to where they started from. So I took the percentage change of the marks for each individual student, and then averaged the percentage change (Diff_1_3Pct:

| IsTutored | Students | Mark1     | Mark3     | Diff_1_3  | Diff_1_3Pct |
|-----------|----------|-----------|-----------|-----------|-------------|
| FALSE     | 68       | 60.863636 | 54.986842 | -3.573529 | -0.040333   |
| TRUE      | 31       | 58.6875   | 68.69697  | 9.516129  | 0.258208    |

I'm wondering about a few things:

  • Is percentage change a good way to approach this?
  • Because I take the percentage change of each student's mark and then average these to arrive at Diff_1_3Pct, Diff_1_3Pct is not the same as the percentage change of the average of Mark1 and the average of Mark3. For example, non tutored students dropped about 6 points from 60.8 to 54.9. This is a drop of about 10%. But Diff_1_3Pct is about -4%. Is this Simpson's Paradox?
  • I could instead take the percentage difference of the average of Mark3 and the average of Mark1 (instead of the percentage difference of the individual scores), however if I did that, I'm not sure how I could calculate the confidence intervals

-- UPDATE -- At the suggestion of user158565, I ran a mixed linear model.

data.head()

+-----------+------+-----------+------+
| StudentID | test | isTutored | Mark |
+-----------+------+-----------+------+
| 2217214.0 | 0    | 1         | 81.0 |
+-----------+------+-----------+------+
| 2206910.0 | 0    | 0         | 51.0 |
+-----------+------+-----------+------+
| 2212272.0 | 0    | 1         | 44.0 |
+-----------+------+-----------+------+
| 2207991.0 | 0    | 0         | 72.0 |
+-----------+------+-----------+------+
| 2215861.0 | 0    | 1         | 50.0 |
+-----------+------+-----------+------+

Model:

import statsmodels.api as sm
import statsmodels.formula.api as smf

data = tutormarks_reshape[tutormarks_reshape.CourseNumber == '24414']
model = sm.MixedLM.from_formula("Mark ~ test*isTutored",data,groups=data["StudentID"])
result = model.fit()
print(result.summary())


Results:
           Mixed Linear Model Regression Results
===========================================================
Model:              MixedLM  Dependent Variable:  Mark     
No. Observations:   198      Method:              REML     
No. Groups:         99       Scale:               198.8986 
Min. group size:    2        Likelihood:          -847.6510
Max. group size:    2        Converged:           Yes      
Mean group size:    2.0                                    
-----------------------------------------------------------
                Coef.  Std.Err.   z    P>|z|  [0.025 0.975]
-----------------------------------------------------------
Intercept       64.919    3.154 20.586 0.000  58.738 71.100
test            -1.787    1.209 -1.477 0.140  -4.157  0.583
isTutored      -10.064    5.636 -1.786 0.074 -21.110  0.981
test:isTutored   6.545    2.161  3.028 0.002   2.309 10.781
groups RE      179.006    4.009                            
===========================================================
$\endgroup$
  • $\begingroup$ Do you have scores from each student at each test? Can you perform linear mixed model? $\endgroup$ – user158565 Nov 20 '18 at 18:37
  • $\begingroup$ Yes, I have the scores from each student for each test. $\endgroup$ – Michael C Nov 20 '18 at 18:51
0
$\begingroup$

Mixed linear model is the best choice. Repose variable is score, fixed effects are regression coefficients of (1) is tutored or not (2) test 1 or test 3 and (3) interaction between (1) and (2). Random effect is student specified random intercepts. Random intercept is used to incorporate the 2 repeated tests on the same student. The fixed effect interaction between (1) and (2) is what you want. ( the students who received tutoring improved more (or declined less) than students who did not receive tutoring.)

Data frame:

 studentID   test   isTutored     score
 ----------------------------------------    
    1         0         0           65
    1         1         0           48
    2         0         1           50
    2         1         1           63
  .....................................
   99         1         0           88
  --------------------------------------

test = 0 means Mark1 and =1 means Mark3.

Based on your model fitting, we have following estimated mean scores

 Mark1, not tutored     64.919
 Mark1, tutored         64.919-10.064
 Maek3, not tutored     64.919- 1.787
 Mark3, tutored         64.919 - 1.787 - 10.064 + 6.545

The change is -1.787 for not tutored and -1.787 + 6.545 for tutored. So difference of changes is 6.545.

test:isTutored   6.545    2.161  3.028 0.002   2.309 10.781

The p = 0.002 indicate that there is difference of changes.

$\endgroup$
  • $\begingroup$ I'm using python statsmodels. Could you help me interpret the results (I added them to the question above)? $\endgroup$ – Michael C Nov 20 '18 at 20:47
  • $\begingroup$ See the last part I added to the Answer. $\endgroup$ – user158565 Nov 20 '18 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.