I created a software platform on which about 110 participants could contribute at any time of their choosing over the course of 12 weeks (think something like a wiki). These participants were then randomly split to use one of two otherwise identical platforms. One key difference was implemented between the two platforms, which I hypothesized to cause a different participation rate.
That means I have participation data over the course of 12 weeks (down to the second) for 2 groups of ~50 people. The participation rate is highly variable - some folks don't participate for 4 weeks at a time and then participate heavily at the end. Others participate in the beginning and then never again.
Independent-samples t-tests show differences between the conditions in terms of total number of times participating. But I don't think that's the most informative. I want be able to run a statistical test to show that the difference I implemented led to an increased participation rate over time.
Some tricky parts of this:
- Because it's a collaborative software platform, the cases within each condition are technically non-independent. Users within each condition could participate with each other, although they generally did not (<10 of the ~6000 participation events involved working with someone else). Because of this, I'm thinking they are quasi-independent, but I think this is a theoretical argument and not a statistical one (although I'm not sure).
- Users were also assigned to work on replicated projects across the conditions, and doubled within-condition. For example, there was a Project A, Project B, Project C on both platforms, and 2 users on each platform were assigned to each Project (a total of 4 people working on Project A, 2 in the experimental condition and 2 in the control). The two folks within each platform condition still worked independently, except in cases as described immediately above this (tricky part 1).
- Participation within-subject is definitely not independent, and this is definitely a statistical problem. Each time a participation event occurs, the user is working on a single project (again, think of it like a wiki). So when a user participates early, that means their later edits are on that same product.
Things I have considered:
- A chi-square test of independence examining number of participations by week (12 conditions) crossed with condition (2 conditions). But because the weeks are not independent, I'm pretty sure this is an inappropriate analytic strategy.
- A repeated-measures ANOVA examining a within-subject "week" variable by between-subject "condition." But the data is highly non-normal (the count data is heavily positively skewed).
- A hierarchical linear model examining within-subject weekly participation rates, nested within persons, with a person-level condition variable. But the same non-normality problem occurs here (it is still count data).
Is there another approach I should take here? Am I missing something that will handle this better?
Edit: I have log-ins, but didn't think of that as meaningful - these are more meaningful participation events. Here is a graph of the cumulative weekly participation events by group, which might help illuminate what I'm talking about. This is the most meaningful metric I can come up with. Note that in this graph, individuals might be represented several times (if a three users participated 30 times, twice and once during Week 1, that increases Week 1 by 33).