I built a suggestion system, which provides a list of candidate items to users. As users choose some items from the suggested list, the system remembers those choices, and the next time the system reorder the candidate list so the items that were heavily selected before will appear at the top of the list. So the more user use and select items, the better the ranking in the future.

I asked some users to try out the system and collect the time each user spent in selecting items. From my raw calculation, I know that on average, the first half of the users spent 20% more time than the second half of the users. This may suggest that the self-improvement of the system is effective.

But I don't know which statistical test to use in order to check this hypothesis. Some of my problems include:

  • Which distribution to use
  • Since the system is changing as more and more users try it, how to deal with this?
  • I only have 12 users now, it is not that many, how to deal with this?
  • 1
    $\begingroup$ Just as a remark, there are other StackExchange sites which might be better suited for this (I think there are statistic, data collection, as well as programming sites of course). Nevertheless, it's your own choice, and Math has of course a high visiting rate. $\endgroup$ – Nick Kidman Aug 8 '12 at 9:26
  • $\begingroup$ Might be better addressed at stats.se. If you're comparing the time taken (as a measure of performance) between the first six and last six users, the Wilcoxon Signed Rank Test is probably a good place to start. $\endgroup$ – mstrfrdmx Aug 8 '12 at 10:02

Since you sound like you have data from individuals over time, you should not ignore that fact; this is something akin to analysis of growth curves.

In any case, it's longitudinal and you probably would use some hierarchical model with mixed effects, though the very small number of users may mean this might be more suitable later, after the numbers have grown some.

One possibility to consider is the rate (speed) rather than the time. It may be 'nicer' to model.

Further details would make this easier to give more extensive answers on.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.