Scoring items which are not easily compared

Question

First of all, I apologize since this question has probably been asked many times and is easily answered. However, as a statistics amateur I simply couldn't figure out what keywords are relevant to my question.

Suppose you have 100 merchants and 100 products. Each merchant sells a certain range of products, ranging from only one product to all 100 products. Also, products are sold in widely different proportions, which differ among merchants, and are subject to the merchant's individual (irrational) preferences.

Whenever a merchant makes a "pitch" on the market, we observe whether or not he manages to sell the product he's pitching. We assume the probability of success depends (a) on the skill of the merchant and (b) the attractiveness of the product. The products' prices are fixed, so that's not a factor.

The data we have consists of millions of pitches. For each pitch, we know whether or not it was successful, the merchant, and the product.

Obviously, if we compare merchants by their average success rate, this information is useless because every merchant sells different products. Likewise, if we compare products, we gain no information since every product is sold by different merchants.

What we want is a skill score for each merchant, which is independent of the products the merchant is selling, and an attractiveness score for each product, which is independent of the merchants who are selling it.

I don't need a comprehensive explanation, just some keywords to point me in the right direction. I literally have no idea where to start.

Edit: Note that our assumption is that the product attractiveness is merchant-independent and the merchant skill is product-independent, i.e. there are no merchants which are better at selling certain products but worse at selling others.

Answer 1

Let me expand on alternative solution proposed by @curious_cat.

$P_{ij}$ is the matrix of pitches

$L_{ij}$ is the matrix of sells

$S_{ij} = L_{ij}/P_{ij}$ is the matrix of success rates (elementwise division where it exists and 0 elsewhere)

As @curious_cat suggested, you want to approximate $S_{ij}$ by the outer product of two positive vectors

$$S_{ij} \approx M_i \times A_j^T$$

Least square minimization will lead to

$$\min | S_{ij} - M_j \times A_i^T |_2$$ where $| \quad |_2$ is the Frobenius norm.

BUT you do not want to minimize for the entries in which $S_{ij}$ is not defined. So what you realy want is something like:

$$ \min |W_{ij} \odot (S_{ij} - M_j \times A_i^T)|_2$$ where $\odot$ is the elementwise multiplication.

1) At a first approximation, $w_{ij}$ is 0 where $p_{ij}$ is 0 and 1 elsewhere.

This is a weighted non-negative matrix factorisation (or approximation) problem. Google should give some references to it.

2) Now, shooting from the hip, let us try to answer the point also made by @curious_cat that you should trust more a success rate of 1000 sells over 2000 pitches than a 2 sells over 4 pitches.

The weight $w_{ij}$ need not to be uniformly 1 for the entries that are defined in $S_{ij}$. One can give it more weight to success rates with higher pitches.

My guess is to use $\sqrt{p_{ij}}$ as the weight. The intuition is that the confidence interval on the success rate is inversely proportional to $\sqrt{p_{ij}}$.

Answer 2

This type of problem is typically referred to in econometrics and marketing research as a "choice modeling" problem. Texts dealing with such problems include: Louviere, J., D. A. Hensher, et al. (2000). Stated Choice Methods: Analysis and Application. Cambridge, Cambridge University Press. Train, K. E. (2009). Discrete Choice Methods with Simulation. Cambridge, Cambridge University Press. Rossi, P. E., G. M. Allenby, et al. (2005). Bayesian Statistics and Marketing, Wiley.

The simplest practical model you could estimate would be a binary logit model with the dependent variable indicating when an object is purchased versus when it is not purchased, with two independent variables: a categorical variable for merchant and a categorical variable for product. (Or, if you do not know anything about when a product is not purchased, you could use Poisson regression or some other counts model.)

The parameter estimate for each merchant would be their skill score and the parameter for each product would be the "attractiveness score". The "attractiveness" score is more commonly referred to as a "utility" in choice modeling.

A practical computational problem you will experience is that unless you have only a few hundred merchants and few hundred categorical variables you will struggle to estimate the model and may need a "random effects" model (sometimes referred to as a "hierarchical model" in this context).

In addition to the assumption that you mention, a key set of assumptions that will determine the validity of your analysis will relate to which alternatives are available at a given time. For example, a product that is intrinsically unattractive may be purchased regularly because the more attractive products are not available at the purchase time. This effect can have a very large impact upon your resulting estimates, as when it is ignored you inadvertently will confound the attractiveness of a product with its availability. The texts cited earlier discuss various modifications of choice models to deal with many of the types of assumptions likely relevant to your problem.

Answer 3

Your problem can be modeled by a Rasch Model. Here is a document that explains the model with the following example

Rasch model is a statistical model of a test that attempts to describe the probability that a student answers a question correctly. It assigns to every student a real number, a, called the "ability", and to every questions a real number, d, called the "difficulty".

This is similar to your situation where each merchant has some inherent "skill" and each product has an inherent "attractiveness".

Answer 4

Why not for each merchant compute a success rate for every product he sells $S_{ij}$. ($i$ indexes products and $j$ indexes merchants) Average this and compute a merchant average baseline success rate($S_j$). Now compute differences ($\delta S_{ij}=S_{ij} - S_j$). Each of this $\delta S_{ij}$ indicates how much better or worse every product does with respect to that merchants baseline success rate.

If you sum up this $\delta S_{ij}$ over all merchants j you'd obtain some sort of score of the attractiveness of every product $S_i$?

The merchant skill metric would be a dual of this. One problem is this doesn't weigh in the confidence level motivated by large data. i.e. 2 successes out of 4 pitches ought to (perhaps) matter less than 1000 successes out of 2000 pitches? You'd have to find some way to adjust for that in case it matters.

Alternatively: Assume every merchant has a skill value $M_j$ and every product has a product attractiveness $A_i$. You could model the success rate of product $i$ sold by merchant $j$ ($S_{ij}$) as some function of $M_j$ and $A_i$ with possible cross terms. If you fit this you might be able to score using the coefficents.

If you consider $S_{ij} = M_j \times A_i + \epsilon_{ij}$ you get one simple model. The matrix of success elements is possibly sparse (since not all merchants sell all products). If it were indeed fully populated you must estimate 200 coefficients from 100x100 success rate numbers such that you minimize $\epsilon_{ij}$ in some sort of least squares sense.

Possible flaws:

I don't see an easy way to interpret relative scores. e.g. If two Products have an attractiveness of $A_{i1}$ and $A_{i2}$ how much better is one than the other? A simple ratio? A log likelihood? etc. Perhaps there is some interpretation but I cannot see it yet. From a strictly ordering perspective it shouldn't matter.

PS How sparse is your matrix? Knowing that you have millions of pitches maybe not too sparse? Or is it? i.e. Out of a maximum possible 10,000 merchant-product combinations how many are filled (i.e. have at least one pitch)?

PS1 Uniqueness. I cannot prove whether your $M_j$ and $A_i$ values will be unique or even close to. If there are multiple solutions it'll be an interesting situation. Maybe there are stronger math results about this?

Answer 5

I think you are looking to attribute qualities that are not inherent in, or do not follow from, your data. You have unambiguous data on success rate, and there should be a way to calculate or estimate a merchant's "adjusted success rate" given the rate at which his products tend to sell among all merchants. Similarly, there should be a way to determine each product's adjusted success rate given the success rates of the merchants who tend to sell it. These two angles on the analysis might be accomplished with a nested/hierarchical/multi-level logistic regression, if the data are suitable for it. But that wouldn't necessarily reveal the attributes of "skill" or "attractiveness"; it might yield workable proxies for them, but how adequate these proxies would be is a substantive question more than a statistical one.

Answer 6

I would just create a 2 way table for this. For e.g. rows corresponding to different merchants and columns corresponding to different products. Each cell in this 100 x 100 table/matrix represents counts/proportion for no. of times the combination was successful.

Once this is done, you can sort this matrix by rows and then by columns (or the other way round) to get the product and merchant skills ordering.

Answer 7

I'd recommend a logistic regression with merchants and products as random effects. In R, this would look like:

library("lme4")
fit <- glmer(sold ~ (1|merchant) + (1|product), data, family=binomial, REML=TRUE, verbose=TRUE, weights)
summary(fit)
ranef(fit)

Extracting the estimates is relatively straightforward, and I handle millions of data points with approaches similar to this on standard workstations all the time. The model fitting typically only takes a few minutes.