# Creating an index based on a set of measurements without a target for purpose of rank ordering

The problem I'm trying to solve here is very simple but the available data is very limited. That makes it a hard problem to solve.

The available data are as follows:

1. I have 100 patients and I need to rank order them in terms of how healthy they are.
2. I only have 5 measurements for each patient. Each of the five readings is coded as a numeric value, and the rule is that the bigger the reading the healthier is the patient.

Should I have some sort of doctor's "expert judgement based ranking" I could use that as the target variable and fit some sort of an ordinal logistic regression model trying to predict doctor's assessment. However, I don't have that. The only thing I have is (1) and (2).

How would you come up with a simple "scoring" algorithm which would combine those five measurements into a single score which would be good enough (not perfect) in rank ordering patients?

A simple approach would be to calculate the sum score or the mean. Another approach would not assume that all variables are of equal importance and we could calculate a weighted mean.

Let's assume we have the following 10 patients and variables v1 to v5.

> set.seed(1)
> df <- data.frame(v1 = sample(1:5, 10, replace = TRUE),
+                  v2 = sample(1:5, 10, replace = TRUE),
+                  v3 = sample(1:5, 10, replace = TRUE),
+                  v4 = sample(1:5, 10, replace = TRUE),
+                  v5 = sample(1:5, 10, replace = TRUE))
>
> df
v1 v2 v3 v4 v5
1   2  2  5  3  5
2   2  1  2  3  4
3   3  4  4  3  4
4   5  2  1  1  3
5   2  4  2  5  3
6   5  3  2  4  4
7   5  4  1  4  1
8   4  5  2  1  3
9   4  2  5  4  4
10  1  4  2  3  4


1. Sum score and ranks

> df$sum <- rowSums(df) > df$ranks <- abs(rank(df$sum) - (dim(df)[1] + 1)) > df v1 v2 v3 v4 v5 sum ranks 1 2 2 5 3 5 17 4.0 2 2 1 2 3 4 12 9.5 3 3 4 4 3 4 18 2.5 4 5 2 1 1 3 12 9.5 5 2 4 2 5 3 16 5.0 6 5 3 2 4 4 18 2.5 7 5 4 1 4 1 15 6.5 8 4 5 2 1 3 15 6.5 9 4 2 5 4 4 19 1.0 10 1 4 2 3 4 14 8.0  2. Mean score and ranks (note: ranks and ranks2 are equal) > df$means <- apply(df[, 1:5], 1, mean)
> df$ranks2 <- abs(rank(df$mean) - (dim(df)[1] + 1))
> df
v1 v2 v3 v4 v5 sum ranks means ranks2
1   2  2  5  3  5  17   4.0   3.4    4.0
2   2  1  2  3  4  12   9.5   2.4    9.5
3   3  4  4  3  4  18   2.5   3.6    2.5
4   5  2  1  1  3  12   9.5   2.4    9.5
5   2  4  2  5  3  16   5.0   3.2    5.0
6   5  3  2  4  4  18   2.5   3.6    2.5
7   5  4  1  4  1  15   6.5   3.0    6.5
8   4  5  2  1  3  15   6.5   3.0    6.5
9   4  2  5  4  4  19   1.0   3.8    1.0
10  1  4  2  3  4  14   8.0   2.8    8.0


3. Weighted mean score (i.e. I assume that V3 and V4 are more important than v1, v2 or v5)

> weights <- c(0.5, 0.5, 1, 1, 0.5)
> wmean <- function(x, w = weights){weighted.mean(x, w = w)}
> df$wmeans <- sapply(split(df[, 1:5], 1:10), wmean) > df$ranks3 <- abs(rank(df$wmeans) - (dim(df)[1] + 1)) > df v1 v2 v3 v4 v5 sum ranks means ranks2 wmeans ranks3 1 2 2 5 3 5 17 4.0 3.4 4.0 3.571429 2.5 2 2 1 2 3 4 12 9.5 2.4 9.5 2.428571 9.0 3 3 4 4 3 4 18 2.5 3.6 2.5 3.571429 2.5 4 5 2 1 1 3 12 9.5 2.4 9.5 2.000000 10.0 5 2 4 2 5 3 16 5.0 3.2 5.0 3.285714 5.0 6 5 3 2 4 4 18 2.5 3.6 2.5 3.428571 4.0 7 5 4 1 4 1 15 6.5 3.0 6.5 2.857143 6.0 8 4 5 2 1 3 15 6.5 3.0 6.5 2.571429 8.0 9 4 2 5 4 4 19 1.0 3.8 1.0 4.000000 1.0 10 1 4 2 3 4 14 8.0 2.8 8.0 2.714286 7.0  • Thanks. My measurements are on different scale. For example v1 goes from 0 to 5 v2 goes from 1000 to 2000 etc.... How do you recomend dealing with that? Jun 3, 2011 at 7:43 • (1) You really want to update your question. More experienced people might come up with better solutions since I am really not an expert for this kind of questions. (2) You could standardize your measures (z-score) and then proceed as explained above. (3) Are your measures correlated? Do they measure the same theoretical construct. If yes, you could run sth like a factor analysis. Jun 3, 2011 at 9:46 • @bremd Is there any "quantitative" method of coming up with weights ... I'm primarily thinking of using correlation and or multicolinearity as a valuable source of information? Jun 29, 2011 at 7:44 • @user333 I am sorry but I am not aware of such a method. I had theoretical weights in mind... Jun 29, 2011 at 10:35 Any function$f: \mathbb{R}^5 \to \mathbb{R}$that is separately increasing in each of its arguments will work. For example, you can select positive parameters$\alpha_i$and any real parameters$\lambda_i$and rank the data$(x_1, x_2, x_3, x_4, x_5)\$ according to the values of

$$\sum_{i=1}^{5} \alpha_i (x_i^{\lambda_i} - 1) / \lambda_i \text{.}$$

Evidently some criterion is needed to select among such a rich set of distinctly different scores. In particular, the simple obvious solutions (frequently employed, unfortunately) of just summing the scores or first "normalizing" them in some fashion and then summing them will suffer from this lack of grounding in reality. To put it another way: any answer that does not derive its support from additional information is a pure fabrication.

Because this problem is essentially the same as Creating an index of quality from multiple variables to enable rank ordering, I refer you to the discussion there for more information.

I would just simply sum them up, weighting each factor if necessary.

• The scale of the measurements is very different... Would need to normalize first! Jun 2, 2011 at 22:26
• How do you recommend I derive weightings Jun 2, 2011 at 22:27
• @user333 Wouldn't you agree that the information that the "scale of the measurements is very different" is an important information? You might want to update your question. Jun 2, 2011 at 22:40

How about generating a synthetic binary target variable first and then running a logistic regression model?

The synthetic variable should be something like... "If the observation is in the top decile on all of the input variable distributions flag it as 1 else 0"

Having generated the binary target variable... Run logistic regression to come up with probabilistic metric 0 to 1 assesing how far/close in the tails of multiple distributions observation is?

• @user That sounds like a sophisticated way to cover up the fact that you are making up the answer!
– whuber
Jun 8, 2011 at 19:00
• How do you mean that? The data will speak for itself... I'm more concerned about the methodology... Is it sensible thing to do Jun 8, 2011 at 20:07
• @user There are limits to what data can tell you. From the original description, they contain no information concerning which of the five factors (or combinations thereof) deserve the greatest weights when assessing overall health. Of course you have partial ranking information given by dominance: when all the readings of one patient exceed the corresponding readings for another, the first patient is healthier. But you have to get beyond this and that involves calculating trade-offs. Logistic regression won't do that for you.
– whuber
Jun 8, 2011 at 20:12
• Thanks for good comment. What exactly do you mean by calculating trade-offs? Can you give me one example? Jun 8, 2011 at 21:01
• @user Consider two specific readings per patient, one on a scale from 0 to 10 and another on a scale from 0 to 100. If a patient with readings of (10, 50) is as healthy as a patient with readings of (5, 70), then 10-5 = 5 units of the first reading have been traded off for 70-50 = 20 units of the second. The questions to answer are (1) are the trade-offs constant across levels; e.g., is a (0, 90) also as healthy as a (5, 70); and (2) does the amount of trade-off depend on the levels of the third, fourth, and fifth variables?
– whuber
Jun 8, 2011 at 22:31