1
$\begingroup$

My situation: I am assessing outcomes of a university course redesign. One of the goals of the redesign project was to increase successful course completion, as measured by a decrease in the course DFW rate. (The DFW rate refers to the proportion of students who earn a D, fail, or withdraw.) The redesign was done in phases, and I want to compare the average DFW rates for Phase 1 (the original course), Phase 2 (first redesign), and Phase 3 (second redesign) and estimate the effect size and confidence intervals.

My dilemma: I am unsure what test(s) and effect size measures are okay to use with aggregated data, a dependent variable that is a proportion, and, if needed, enable me to account for the fact that each group (Phase) consists of a different number of course sections with a different number of students in each section. What would be the least complicated (but still an appropriate) way to go?

My dataset: My dataset is aggregated by course section. There are 33 records, one for each of the on-campus sections taught by a single professor between the Fall of 2005 and Fall of 2013. Aggregate stats are based on all the students in these sections who were taking the course for a letter grade (i.e., not pass/fail or audit credit) and were taking it for the first time.

Here is a summary produced in Stata:

. codebook, compact dots

Variable   Obs Unique      Mean       Min       Max  Label
-------------------------------------------------------------------------------------------
PHASE       33      3  2.030303         1         3  CR Phase
YEAR        33      9  2009.939      2005      2013  Year
SEMESTER    33      2  .3939394         0         1  Semester (0=Fall, 1=Spr)
TERM        33     17  9.484848         0        16  Term (0=Fall 05, 1=Spr 06, etc.)
SECTION     33      3  1.757576         1         3  Section
STRM_SECT   33     33         .         .         .  Term Code & Section
DFW_RATE    33     32  .1364054  .0243902  .2727273  Proportion of Students with D, F, or W
DFW_SD      33     32  .3324413  .1552067  .4494666  Std. dev. of DFW (0=ABC, 1=DFW)
STUDENTS    33     27  76.33333        41       123  # Students
PHASE1      33      2  .2121212         0         1  Phase 1
PHASE2      33      2  .5454545         0         1  Phase 2
PHASE3      33      2  .2424242         0         1  Phase 3
-------------------------------------------------------------------------------------------

Here are the data, themselves:

by PHASE, sort: list, clean

    PHASE   YEAR   SEMESTER      TERM   SECTION   STRM_SECT  DFW_RATE   DFW_SD   STUDENTS   PHASE1   PHASE2   PHASE3 
  1.    1   2005       Fall   Fall 05       003     2058_3   .18085     .38696         94        1        0        0  
  2.    1   2006     Spring   Spr. 06       001     2061_1   .23288     .42559         73        1        0        0  
  3.    1   2006       Fall   Fall 06       002     2068_2   .125       .33261         88        1        0        0  
  4.    1   2007     Spring   Spr. 07       001     2071_1   .19767     .40058         86        1        0        0  
  5.    1   2007       Fall   Fall 07       002     2078_2   .06667     .25112         75        1        0        0  
  6.    1   2008     Spring   Spr. 08       001     2081_1   .2         .40212         95        1        0        0  
  7.    1   2008       Fall   Fall 08       003     2088_3   .17204     .37946         93        1        0        0  
------------------------------------------------------------------------------------------------------------------------
  8.    2   2008       Fall   Fall 08       001     2088_1   .17391     .38322         46        0        1        0  
  9.    2   2008       Fall   Fall 08       002     2088_2   .08333     .27784         96        0        1        0  
 10.    2   2009     Spring   Spr. 09       001     2091_1   .25        .43693         56        0        1        0  
 11.    2   2009     Spring   Spr. 09       002     2091_2   .27273     .44947         55        0        1        0  
 12.    2   2009       Fall   Fall 09       001     2098_1   .07547     .26668         53        0        1        0  
 13.    2   2009       Fall   Fall 09       002     2098_2   .16129     .36979         93        0        1        0  
 14.    2   2009       Fall   Fall 09       003     2098_3   .11881     .32518        101        0        1        0  
 15.    2   2010     Spring   Spr. 10       002     2101_2   .19643     .40089         56        0        1        0  
 16.    2   2010       Fall   Fall 10       001     2108_1   .04082     .19991         49        0        1        0  
 17.    2   2010       Fall   Fall 10       002     2108_2   .10345     .30631         87        0        1        0  
 18.    2   2011     Spring   Spr. 11       001     2111_1   .15217     .36316         46        0        1        0  
 19.    2   2011       Fall   Fall 11       001     2118_1   .17204     .37946         93        0        1        0  
 20.    2   2011       Fall   Fall 11       002     2118_2   .13008     .33777        123        0        1        0  
 21.    2   2011       Fall   Fall 11       003     2118_3   .05172     .2234          58        0        1        0  
 22.    2   2012     Spring   Spr. 12       001     2121_1   .07317     .26365         41        0        1        0  
 23.    2   2012     Spring   Spr. 12       002     2121_2   .12676     .33507         71        0        1        0  
 24.    2   2012       Fall   Fall 12       001     2128_1   .21212     .41194         66        0        1        0  
 25.    2   2012       Fall   Fall 12       002     2128_2   .15942     .36875         69        0        1        0  
------------------------------------------------------------------------------------------------------------------------
 26.    3   2010     Spring   Spr. 10       001     2101_1   .13793     .34784         58        0        0        1  
 27.    3   2011     Spring   Spr. 11       002     2111_2   .12245     .32949         98        0        0        1  
 28.    3   2012       Fall   Fall 12       003     2128_3   .08654     .28252        104        0        0        1  
 29.    3   2013     Spring   Spr. 13       001     2131_1   .11905     .32777         42        0        0        1  
 30.    3   2013     Spring   Spr. 13       002     2131_2   .10127     .30361         79        0        0        1  
 31.    3   2013       Fall   Fall 13       001     2138_1   .1413      .35024         92        0        0        1  
 32.    3   2013       Fall   Fall 13       002     2138_2   .0396      .196          101        0        0        1  
 33.    3   2013       Fall   Fall 13       003     2138_3   .02439     .15521         82        0        0        1  

My initial notes & thoughts: I still have so much to learn and would appreciate your help in correcting any misconceptions and flawed logic in my attempt to address the question thus far.

I'm thinking weights probably ought to be applied to the data since each section's DFW_RATE is derived from a different number of students' grades. From my reading (primarily the Stata help docs/base reference in this case), it seems "analytic weights" based on the variable STUDENTS would be most appropriate. (As far as I can tell, SPSS only gives you the option to use "frequency weights") I'm not certain if/how the standard deviations (DFW_SD) associated with the section DFW rates should be used.

A one-way ANOVA with analytic weights seems like it would be the simplest (even if a bit simplistic) way to test for overall statistical significance using the current dataset. I could then follow-up with Bonferroni-corrected post-hoc comparisons of the means (or should it be proportions?) of PHASE 1 vs 2, PHASE 2 vs 3, and PHASE 3 vs 1. I may be overlooking something obvious or doing something incorrectly, but I'm not seeing any major violations of the ANOVA assumptions: - Despite being bounded between 0 and 1 and having values generally below .2, DFW_RATE is approximately normally distributed at each level of PHASE (per Shapiro-Wilk test) - Per Levene's test, no evidence of heterogenous variances - The group sizes are unequal, which is a consideration but not a crucial assumption - Regarding the independence of observations assumption, no student is in more than one section (course repeats were removed prior to data aggregation) and all students had the same instructor. The section number (SECTION) is associated neither with PHASE nor with DFW_RATE, and there is no apparent statistical dependence between time (TERM) and DFW_RATE among sections taught using the same course design (same PHASE).

However, ANOVA treats PHASE as nominal and not ordinal (to what extent is this problematic?).

In terms of which effect size measures to use with ANOVA, would either eta-squared or omega-squared work? For the omnibus test and post-hoc comparisons, do weights need to be applied somehow when calculating effect size and CIs?

My software: I have been working in Stata v.13.1 but have SPSS 22.0 as well and have R installed too (though am less familiar with it). My gratitude: Thanks in advance for your help!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.