How to estimate loyalty to service?

Question

Let's say there are two services $A$ and $B$ that solve same problem. We would like to know which one users more loyal to. Loyalty $L(X)$ is probability that user will use service $X$ when asked to solve problem.

Well for every user we can estimate $L$ for service $A$ like this $L(A) = \frac{N(A)}{N(A) + N(B)}$ where $N(X)$ is number of uses of $X$. We can even smooth it with Laplacian smoothing and get $L'(A) = \frac{N(A) + 1}{N(A) + N(B) + 2}$ but it's not the case.

In reality 20% of users have $N(A) + N(B) = 1$, then there are two types of $L(A)$: 0 and 1 both with variance like 0.5 =).

Maybe there are known methods how to behave in such situations when there was almost none experiments conducted. We need to have better estimation on both $L$ and it's variance.

Answer 1

As @MichaelChernick says, there is probably no good answer to reliably estimate the user's loyalty, for users who have not used any service very often.

(Arguably it would be reasonable to classify those users as not very loyal. You might decide that a user must use your service a minimum number of times before you're prepared to consider them a loyal user.)

One approach to dealing with these issues might be to use a Bayesian method. Compute the average loyalty, across all users. For a user who has no uses, the average loyalty is your best estimate. As the number of uses increases, your estimate should start to move in the direction of your $L(A)$ value.

In particular, the following might be a reasonable estimate: $$L_\text{est}(A) = \frac{n}{n+n^*} L(A) + \frac{n^*}{n+n^*} L^*(A), $$ where $n=N(A)+N(B)$ is the number of times this user has used a service, $n^*$ is the average value of $n$ across all users, $L(A)=N(A)/(N(A)+N(B))$ is your naive (uncorrected) loyalty ratio for this user, and $L^*(A)$ is the average value of $L(A)$ across all users.

If you know something about the characteristics of the user, and you think these characteristics will have a significant effect on user loyalty, you could perhaps take the average over all other users who are similar to this user, rather than over all users.

Answer 2

This is a small sample size problem. There is no statistical magic that will fix it. To estimate loyalty for an individual user you need several examples where the the user selected from A and B to do the job. Then loyalty customers.to A might be modeled as the success proportion parameter for a binomial distribution where success means that A is selected. So the estimation problem can be easily dealt with as long as N(A)+N(B) is reasonably large (say at least 10). But for N(A)+N(B)<5 it is folly to try to make such an estimate. The only way I can see dealing with the small total sample size would be to pool customers into groups based on similar characteristics for these customers.