Ranking of normalized variables

Question

I'd like to rank 100 objects according to 20 variables. I chose a weight for each of this variable. As these variables can have very different values (some are a percentage while others are real numbers), I normalized them (mean of 0 and variance of 1) and I computed a weighted sum which was my ranking.

I saw that the top 14 objects stood very apart from the other objects so I did the normalization and the weighted sum with only these 14 objects. The new ranking I got was somewhat different from the first one. It is: 1, 2, 4, 3, 9, 6, 7, 5, 8, 11, 10, 14, 13, 12 (so, the object at position 9 in the first ranking is at position 5 in the second ranking). The (Kendall) correlation associated to these two rankings is 0.7582.

I can not figure out why there is an observable difference between these two rankings. My first guess was that the number of objects is much smaller in the second case so the normalization is biased but I can't think of a statistical proof of it.

I thought of changing values so that they follow a t-student distribution (usually used when there is few data) but can not figure out how to do it (and if there is a statistical sense to do it).

P.S.: I didn't include any data because it would be too long to write them out but they are not "private" so I can give them if needed.

Answer 1

It's not impossible to build a scenario where this would happen.

Consider even a simplified case: you have two variables, $A$ and $B$.

• $B$ is $\pm 1$ with probability $\frac{1}{2}$ each.

• For the top 14 objects, $A$ is $\pm 1000000000$ with probability $\frac{1}{2}$ each.

• For the bottom 86 objects, $A$ is $\pm 1$ with probability $\frac{1}{2}$ each.

So, when you:

1. Normalize to variance 1 over all the dataset

2. Normalize to variance 1 over the top 14

and use the same weights between $A$ and $B$ in both cases, you're effectively changing the relative weight between $A$ and $B$ for the top 14 objects. You indeed might get a different ranking for them.