I built a suggestion system, which provides a list of candidate items to users. As users choose some items from the suggested list, the system remembers those choices, and the next time the system reorder the candidate list so the items that were heavily selected before will appear at the top of the list. So the more user use and select items, the better the ranking in the future.
I asked some users to try out the system and collect the time each user spent in selecting items. From my raw calculation, I know that on average, the first half of the users spent 20% more time than the second half of the users. This may suggest that the self-improvement of the system is effective.
But I don't know which statistical test to use in order to check this hypothesis. Some of my problems include:
- Which distribution to use
- Since the system is changing as more and more users try it, how to deal with this?
- I only have 12 users now, it is not that many, how to deal with this?