26
$\begingroup$

I have four numeric variables. All of them are measures of soil quality. Higher the variable, higher the quality. The range for all of them is different:

Var1 from 1 to 10

Var2 from 1000 to 2000

Var3 from 150 to 300

Var4 from 0 to 5

I need to combine four variables into single soil quality score which will successfully rank order.

My idea is very simple. Standardize all four variables, sum them up and whatever you get is the score which should rank-order. Do you see any problem with applying this approach. Is there any other (better) approach that you would recommend?

Thanks

Edit:

Thanks guys. A lot of discussion went into "domain expertise"... Agriculture stuff... Whereas I expected more stats-talk. In terms of technique that I will be using... It will probably be simple z-score summation + logistic regression as an experiment. Because vast majority of samples has poor quality 90% I'm going to combine 3 quality categories into one and basically have binary problem (somequality vs no-quality). I kill two birds with one stone. I increase my sample in terms of event rate and I make a use of experts by getting them to clasify my samples. Expert classified samples will then be used to fit log-reg model to maximize level of concordance / discordance with the experts.... How does that sound to you?

Improve this question
$\endgroup$

5 Answers 5

Reset to default
26
$\begingroup$

The proposed approach may give a reasonable result, but only by accident. At this distance--that is, taking the question at face value, with the meanings of the variables disguised--some problems are apparent:

  1. It is not even evident that each variable is positively related to "quality." For example, what if a 10 for 'Var1' means the "quality" is worse than the quality when Var1 is 1? Then adding it to the sum is about as wrong a thing as one can do; it needs to be subtracted.

  2. Standardization implies that "quality" depends on the data set itself. Thus the definition will change with different data sets or with additions and deletions to these data. This can make the "quality" into an arbitrary, transient, non-objective construct and preclude comparisons between datasets.

  3. There is no definition of "quality". What is it supposed to mean? Ability to block migration of contaminated water? Ability to support organic processes? Ability to promote certain chemical reactions? Soils good for one of these purposes may be especially poor for others.

  4. The problem as stated has no purpose: why does "quality" need to be ranked? What will the ranking be used for--input to more analysis, selecting the "best" soil, deciding a scientific hypothesis, developing a theory, promoting a product?

  5. The consequences of the ranking are not apparent. If the ranking is incorrect or inferior, what will happen? Will the world be hungrier, the environment more contaminated, scientists more misled, gardeners more disappointed?

  6. Why should a linear combination of variables be appropriate? Why shouldn't they be multiplied or exponentiated or combined as a posynomial or something even more esoteric?

  7. Raw soil quality measures are commonly re-expressed. For example, log permeability is usually more useful than the permeability itself and log hydrogen ion activity (pH) is much more useful than the activity. What are the appropriate re-expressions of the variables for determining "quality"?

One would hope that soils science would answer most of these questions and indicate what the appropriate combination of the variables might be for any objective sense of "quality." If not, then you face a multi-attribute valuation problem. The Wikipedia article lists dozens of methods for addressing this. IMHO, most of them are inappropriate for addressing a scientific question. One of the few with a solid theory and potential applicability to empirical matters is Keeney & Raiffa's multiple attribute valuation theory (MAVT). It requires you to be able to determine, for any two specific combinations of the variables, which of the two should rank higher. A structured sequence of such comparisons reveals (a) appropriate ways to re-express the values; (b) whether or not a linear combination of the re-expressed values will produce the correct ranking; and (c) if a linear combination is possible, it will let you compute the coefficients. In short, MAVT provides algorithms for solving your problem provided you already know how to compare specific cases.

Improve this answer
$\endgroup$
10
  • 3
    $\begingroup$ @user My recommendations are in the last paragraph: preferably, find a quantitative expression of "quality" in the scientific literature. Barring that, apply MAVT. Both produce a fixed formula independent of the dataset. That assures comparability. $\endgroup$
    – whuber ♦
    Commented Apr 8, 2011 at 16:06
  • 1
    $\begingroup$ @user The way to handle complication is to simplify afterwards. When you avoid appropriate analyses in the name of simplification, you lose the opportunity to get superior results. (Anyway, it is difficult to see how logistic regression would apply to a concept like "quality," which typically is a continuous variable, not a binary one.) $\endgroup$
    – whuber ♦
    Commented Apr 8, 2011 at 16:16
  • 1
    $\begingroup$ @whuber, Couldn't one view this as a problem of making a formative measure based on the available information, in which case summing the Z-scores is not as bad as you make it sound? $\endgroup$
    – Andy W
    Commented Apr 8, 2011 at 17:38
  • 5
    $\begingroup$ @Andy Could you explain what you mean by "formative measure" and "available information"? // I should point out that many measures of soil suitability for agriculture are not even monotonic, much less linear: for instance, a plant might flourish within a range of pH but suffer with pH's beyond this range in either direction. It would be a special circumstance indeed--maybe one involving a narrow range of values--if a simple linear combination of soil characteristics had any objective relationship to agricultural qualities. $\endgroup$
    – whuber ♦
    Commented Apr 8, 2011 at 17:52
  • 2
    $\begingroup$ @Andy Assuming "quality" is a numeric value to be used for ranking soil samples, then definitely the problem is one of discrete decisions: given a pair of attributes $(y_1, \ldots, y_k)$ and $(x_1, \ldots, x_k)$, which has better quality? You are correct that you need to know something about what quality is in order to create the desired combination of the attributes. The approach I have taken supposes you do not have an independent assessment of quality (that would put us into a regression or response surface modeling situation), but you can make these comparisons with reasonable accuracy. $\endgroup$
    – whuber ♦
    Commented Apr 8, 2011 at 19:09
2
$\begingroup$

Anyone looked at Russell G. Congalton 'Review of Assessing the Accuracy of Classifications of Remotely Sensed Data' 1990 ?. It describes a technique known as error matrix for varing matrices, also a term he uses called ' Normalizing data' , whereby one gets all the different vectors and 'normalizes' or sets them to equal from 0 to 1. You basically change all vectors to equal ranges from 0 to 1.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ How does this resolve the current problem, which is not about classification accuracy but instead about combining multivariate attributes into a single scalar rank? $\endgroup$
    – whuber ♦
    Commented Apr 25, 2022 at 17:31
0
$\begingroup$

One other thing you did not discuss is the scale of the measurements. V1 and V5 looks like they are of rank order and the other seem not. So standardization may be skewing the score. So you may be better transforming all of the variables into ranks, and determining a weighting for each variable, since it is highly unlikely that they have the same weight. Equal weighting is more of a "no nothing" default. You might want to do some correlation or regression analysis to come up with some a priori weights.

Improve this answer
$\endgroup$
2
  • $\begingroup$ How can I use correlation analysis to determine weigh? $\endgroup$
    – user333
    Commented Apr 8, 2011 at 16:36
  • $\begingroup$ If you already have a pre-existing overall measure of quality, e.g expert opinions, (or are willing to accept other variables as a proxy for this), you could choose the highest correlated variables and give it the highest weighting. $\endgroup$
    – Ralph Winters
    Commented Apr 8, 2011 at 17:07
0
$\begingroup$

I had a similar problem recently and though I add my approach to the nice answers. I think in order to find a simple way to determine which variable leads to the best ranking. One could transform your problem to a gridsearch approach:

Basically use a combined score for the ranking which is composed as such:

Finel_score = Var1 * A + Var2 * B + Var3 * C ....

Then you can compute the final score with different values for A,B,C (sklearn gridsearch could be used) ... and compare the resulting ranking to an expected ranking (some ground truth is needed to determine the goodness of you ranking). The best parameters result in the weights of your individual variables.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This method (1) presumes a solution has already been developed in terms of the weights $A,B,C,$ etc.; (2) fails to notice that any such linear function is optimized on the boundary of its domain (making a grid search particularly inefficient); and (3) does not appear to solve anything because of the vagueness of the description. $\endgroup$
    – whuber ♦
    Commented Apr 25, 2022 at 17:30
-3
$\begingroup$

Following up on Ralph Winters' answer, you might use PCA (principal component analysis) on the matrix of suitably standardized scores. This will give you a "natural" weight vector that you can use to combine future scores.

Do this also after all scores have been transformed into ranks. If the results are very similar, you have good reasons to continue with either method. If there are discrepancies, this will lead to interesting questions and a better understanding.

Improve this answer
$\endgroup$
8
  • 4
    $\begingroup$ I disagree. While one would likely be interested in the inter-item correlations for curiosity, all of the variables could be orthogonal yet still contribute to quality. For a silly example the soil in Antarctica may have optimal nitrogen content, but I doubt it would suffice as a suitable climate. $\endgroup$
    – Andy W
    Commented Apr 8, 2011 at 18:45
  • $\begingroup$ @Andy W: In that case, all the variables should be weighted equally, and PCA will tell you that. It would also tell you that the leading component only accounts for a relatively small fraction of the overall variability in the scores matrix. $\endgroup$
    – Hans Engler
    Commented Apr 8, 2011 at 20:52
  • 3
    $\begingroup$ I still disagree. It does not tell you if the scores should be weighted equally. Two items could have a positive correlation yet each has opposite relationships to "quality". The inter-item correlations do not necessarily say anything about the unobserved measure in the given context. If quality were a latent variable and the variables were "reflective" of that latent construct that may be true, but that is not the case in this given example. $\endgroup$
    – Andy W
    Commented Apr 9, 2011 at 3:48
  • $\begingroup$ @Andy, I agree with your point if nothing were known about the association of the observed variables with "quality". But the OP wrote "All of [the variables] are measures of soil quality. Higher the variable, higher the quality" implying a positive association throughout. To be more precise: Let $A$ be the $m \times n$ matrix of observations. Consider the first term $\sigma_1 uv^T$ in the singular value decomposition of $A$. If all $n$ variables have the same association with "quality", one expects all $v_j$ to have the same sign. In that case, use multiples of these $v_j$ as weights. $\endgroup$
    – Hans Engler
    Commented Apr 9, 2011 at 21:56
  • 3
    $\begingroup$ I still disagree. Even if the association is expected to be in the same direction this does not mean the indicators should be inherently given any weight based on their inter-item correlation. The shared variance can only say something about the relationship between the indicators. Think of a regression model in which we predict a known measure of quality from these indicators. The inter-item correlations between the indicators do not tell you what the expected slopes will be. $\endgroup$
    – Andy W
    Commented Apr 10, 2011 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.