Best way to compare software usage data over time across independent conditions?

Question

I created a software platform on which about 110 participants could contribute at any time of their choosing over the course of 12 weeks (think something like a wiki). These participants were then randomly split to use one of two otherwise identical platforms. One key difference was implemented between the two platforms, which I hypothesized to cause a different participation rate.

That means I have participation data over the course of 12 weeks (down to the second) for 2 groups of ~50 people. The participation rate is highly variable - some folks don't participate for 4 weeks at a time and then participate heavily at the end. Others participate in the beginning and then never again.

Independent-samples t-tests show differences between the conditions in terms of total number of times participating. But I don't think that's the most informative. I want be able to run a statistical test to show that the difference I implemented led to an increased participation rate over time.

Some tricky parts of this:

1. Because it's a collaborative software platform, the cases within each condition are technically non-independent. Users within each condition could participate with each other, although they generally did not (<10 of the ~6000 participation events involved working with someone else). Because of this, I'm thinking they are quasi-independent, but I think this is a theoretical argument and not a statistical one (although I'm not sure).
2. Users were also assigned to work on replicated projects across the conditions, and doubled within-condition. For example, there was a Project A, Project B, Project C on both platforms, and 2 users on each platform were assigned to each Project (a total of 4 people working on Project A, 2 in the experimental condition and 2 in the control). The two folks within each platform condition still worked independently, except in cases as described immediately above this (tricky part 1).
3. Participation within-subject is definitely not independent, and this is definitely a statistical problem. Each time a participation event occurs, the user is working on a single project (again, think of it like a wiki). So when a user participates early, that means their later edits are on that same product.

Things I have considered:

1. A chi-square test of independence examining number of participations by week (12 conditions) crossed with condition (2 conditions). But because the weeks are not independent, I'm pretty sure this is an inappropriate analytic strategy.
2. A repeated-measures ANOVA examining a within-subject "week" variable by between-subject "condition." But the data is highly non-normal (the count data is heavily positively skewed).
3. A hierarchical linear model examining within-subject weekly participation rates, nested within persons, with a person-level condition variable. But the same non-normality problem occurs here (it is still count data).

Is there another approach I should take here? Am I missing something that will handle this better?

Thanks all.

Edit: I have log-ins, but didn't think of that as meaningful - these are more meaningful participation events. Here is a graph of the cumulative weekly participation events by group, which might help illuminate what I'm talking about. This is the most meaningful metric I can come up with. Note that in this graph, individuals might be represented several times (if a three users participated 30 times, twice and once during Week 1, that increases Week 1 by 33).

Answer 1

Here are a few thoughts:

Characterise participation

• I think the first step is to understand participation at the individual-level.
• How is participation measured at a granular level?
  • a log-in to the site;
  • duration on site for a given log-in;
  • amount of interaction on the site for a given log-in;
• Consider how granular measures of participation can be aggregated into an overall level of participation, both for the whole 12 weeks and for smaller temporal periods of aggregation (e.g., a week).
  • Presumably, there are conceptual reasons to prefer one index of participation over another (e.g., writing up 2,000 words on the site in one log-in over a couple of hours is probably greater participation than 5 log-ins involving only a little bit of tinkering).
  • As you aggregate, you may find that some of the skew is removed from the data
  • You could also consider transformations (e.g., log) to reduce the skew
  • There might also be other indices of participation over and above overall participation (e.g., regularity of participation).

Explore temporal effects on participation

• I'd examine lots of graphs of time on participation based on different levels of temporal aggregation (e.g., by day, by week, by two weeks, by four weeks). It would be interesting to examine group and individual-level analyses to see both the general trend and how individuals differ in the effect of time.
• In terms of a statistical model, perhaps some form of GEE ( see these references on GEE ) would be suitable for modelling the data if the data is similar to counts. But I'd be curious to hear what others have to say. I wonder whether there is a literature on modelling individual level website usage that would be relevant.
• Some form of clustering might also be interesting as a way of clustering different usage patterns.
• You could also explore (and possibly model) any other temporal effects (e.g., when you've assigned projects to particular individuals, and so on).

Assess effect of condition of participation

• Once you have one or more theoretically meaningful measures of overall participation, a simple t-test might be sufficient.
• If you are worried about independence of observations, you could assess this at this point, particularly if you know which individuals might be grouped together (e.g., using something like ICC). A rough approach would simply be to use a more stringent alpha level.

Answer 2

Here are my two cents:

1. From the graphic, it seems that the main difference is in the beginning. try to remove the begenning (until point 5 ot 6 in the x-axis in the graphic) to see if I'm right.

2. Why don't you model it with a poisson (o negative binomial if there is too much zeros)? Your independente variables would be the platafform, conditions and a time effect.

3. Maybe some hierarchical model could help here.