Designing a Simple (a.k.a 'bad') Ranking value from several values of unknown distribution

Question

I am but a humble developer, so please forgive if I butcher terminology of confuse concepts.

I am looking at designing a simple ranking of Average Internet Audience Rank based upon a bloggers audience size (Twitter/Tumblr/Facebook friends) and how many times their content has been shared (i.e. 'Reach').

Here are the metrics I want to combine:

| metric       | weight |    range      | distribution (est.)
-------------------------------------------------------------
| shares       |  1st   |  0 - Millions | power law
| comments     |  2nd   |  0 -thousands | power law
| Klout score  |  3rd   |  0 - 100      | normal , weighted toward lower end
| Twitter fol. |  4th   |  0 - Millions | power law
| LinkedIn con.|  5th   |  0 - 1000     | normal

Goal

• None of the individual metrics will completely crowd out any of the others if it is significantly higher than normal. For example if someone had 10,000,000 twitter followers, but all other values were 0, I would never want them to rank higher than someone who had even 1000 blog post shares.

I know this Multi-Criteria Decision Analysis which I've been reading up on, but operations research is not a field I'm very familiar with. Hopefully you guys can help point me in the right direction.

Answer 1

There are a huge number of potential functions that could work. What you want is probably a weighted sum, but with some transformation of the variables prior to adding them. The choice of transformation and weights is really arbitrary, and a reasonable way to do it is to programme them into a spreadsheet and then alter the transformation and weights until you have something that fits your criteria for a good overall metric.

My suggested approach is as follows:

1. choose (what I will call) a high reference point for each metric. For instance, you could choose 10,000,000 for shares; 10,000 for comments, and so on.
2. For shares, comments and LinkedIn, since you think they follow power laws, you could take logarithms of the metrics which will convert them to a more linear scale.
3. Take a linear transformation of each metric so that it has a value that is 0 when the original value is 0, and 1 when the original value is the high reference point. For shares, this would be $log(shares + 1) / log(10000001)$. (You need to add 1 because log 0 is not defined). For Klout score it would be simply $klout / 100$.
4. Multiply each transformed metric by a weighting and sum them. If you are able to define all your criteria for a good metric mathematically, then you might be able to determine appropriate weightings by solving a system of linear inequalities. But in most cases I would think trial and error using a spreadsheet (like this one) would be easier, as you will probably want to experiment and see what the effects are as you choose the criteria, rather than having rigid criteria in mind from the outset.

In summary:

$score = \sum_{i=1}^n w_i m_i/h_i + \sum_{i=n+1}^N w_i log(m_i + 1) / log(h_i + 1) $

where $m_1 ...m_n$ are the metrics you think are normally distributed and $m_{n+1} ... m_N$ are the metrics you think have a power distribution, $w$ are the weights and $h$ the high reference points.

Some other possible approaches:

1. instead of guessing a high reference point, you could guess the parameters of an exponential or normal distribution that would fit your metrics, then use a standard score (z-score) as your metric. It can be any value on the same order of magnitude as what you think is the maximum value for each metric.

2. If you actually have a load of data already, then you could examine the actual distribution of each metric, and then use the mean and standard deviations of the existing data to calculate standard scores for both the existing and future data. (Your existing data would, in other words, be the reference point rather than picking an arbitrary high number such as 10,000,000).

3. If you already have some data, you could explore applying a technique like weighted principal components analysis, which would look at how the different metrics vary together, and give you a linear combination of the metrics that would explain the maximum variation. But you would still have to transform the variables and choose weights for them beforehand.