1
$\begingroup$

I'm an educator and I'm delving a little further into stats than I normally would to try to determine how effective a new teaching method is. I'd really appreciate some advice on the best approach to take to make sure I'm producing a robust conclusion.

I teach a course online, and I've implemented a change in the course which I think will improve involvement. I'm measuring involvement simply through access stats - so how many resources on the course the student has viewed each day.

I have created an experiment which involves running Course A and Course B, both with identical content, but Course B includes the change in delivery format. So, Course A is the control, and Course B is the alteration.

I have logged access stats for every user for each day of the 5 day course. The standard pattern is that access volume drops over time, and the number of participants accessing each day drops over time, more than halving on average between the first and the last day. The hypothesis is that Course B will produce a higher total volume of access, and a higher number of participants per day, ie. it encourages students to consume more materials throughout the week and drop in more often.

My question is around how to compare the two sets of data and check that the improvement is significant. When participant stats are totalled and graphed, it looks like this. X is the day number, and Y is the percentage of the total participants who have access the course that day.

The Number of participants accessing the course each day

You can see that Course A, the blue line, looks to be an improvement over the red.

I've tried t-tests on each day to determine the difference, but the variance in participation between individuals is very large and n is quite small (22 and 23 each group), so I'm not getting anything significant. Is there a better way to compare these sets of data, and to determine their significance?

I'd really appreciate any pointers anyone might have!

EDIT: I do have the individual data in a table, which I suspect may be useful to see. It looks like so:

Participant access stats by day

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Doing any test at each time point is not really doing justice to your design or dataset. You could use a mixed effect model and compare the profiles over time - the blue one does look to drop over time more. Some people would call these models repeated measures ANOVA. A good first step would be to make a spaghetti plot. These are available in R and Stata and possibly elsewhere.

Improve this answer
$\endgroup$
-1
$\begingroup$

T-test has a lot of assumptions, which you can easily violate render the test invalid. You should use nonparametric tests.

Since you are dealing with dependent (paired) data, the Wilcoxon signed-rank test would be the most appropriate.

I have put the data from the chart into SPSS, but it fails to reject the null hypothesis (p=0.104).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thankyou for the suggestion, that's really interesting. Does this test compare the sets of data as a whole, ie. days 1 to 5, and pairing by day? I'm not sure I need to show that the whole set of data is statistically different, as long as I can show that days 1 to 3 are. Would I be able to perform this test on each day individually, comparing the two measures? $\endgroup$
    – Colin Gray
    Commented Feb 26, 2015 at 14:41
  • $\begingroup$ It pairs the data on a case-by-case basis (day-by-day in your case). You can read this SPSS tutorial for more information: statistics.laerd.com/spss-tutorials/… $\endgroup$
    – alesc
    Commented Feb 26, 2015 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.