1
$\begingroup$

I'm really interested in index scores like the human development index or economic freedom index where they rank things on a 0-100 scale based off of a bunch of different variables (e.g. press freedom, property rights, etc). I would like to do this with z-scores for multiple columns in a Python notebook.

I'm able to calculate the Z scores just fine using scipy:

from scipy.stats import zscore
no_income_data_important_columns_only.apply(zscore)

and get something like:enter image description here I understand these z-scores are telling me how high or low the values are relative to the average. But now I want to be able to understand how high or low each row is across all of the measurements/columns...

I don't really know what to do with all the Z-scores in order to calculate an index or composite score (on a 0 to 100 scale). Do I multiply them or add them together or do something else to aggregate them?

Improve this question
$\endgroup$
1
  • $\begingroup$ How you combine the z-scores depends on the importance of each. If equally important, the just an ordinary average. Otherwise, an appropriately weighted average. I can't immediately imagine a scenario in which multiplication would be useful. $\endgroup$
    – BruceET
    Sep 10, 2020 at 18:53

1 Answer 1

Reset to default
1
$\begingroup$

Suppose you have a composite z-score that is roughly normal with mean approximately 0 and standard deviation approximately 1. Then transforming with the standard normal CDF $\Phi$ (pnorm in R) will give you scores that are approximately uniformly distributed on $(0,1).$ Multiplying by 100 will give you 'index' scores between $0$ and $100.$

Here is an example in R, beginning with z-scores for 250 simulated items.

set.seed(2020)
z = rnorm(250)  # population mean 0 and SD 1 are the default
summary(z);  length(z);  sd(z)
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
-3.056684 -0.739665  0.067159  0.006864  0.710861  3.201632 
[1] 250
[1] 1.120193
u = pnorm(z)    # approx uniformly distributed on (0,1)
summary(u)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.001119 0.229755 0.526771 0.504363 0.761384 0.999317 
x = 100*u       # proposed index scores
summary(x);  sd(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.1119 22.9755 52.6771 50.4363 76.1384 99.9317 
[1] 30.30388

par(mfrow=c(1,2))
 hist(z, prob=T, col="skyblue2", main="Z Scores"); rug(z)
 hist(x, prob=T, col="skyblue2", main="Index Scores"); rug(x)
par(mfrow=c(1,1))

enter image description here

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ +1. Maybe it is a good idea to start your answer right with your comment you've given to the point how the "composite" (averaged) z-score could be obtained first. $\endgroup$
    – ttnphns
    Sep 10, 2020 at 20:33
  • $\begingroup$ I appreciate the answer. I know Python but don't know R so I'm just trying to follow this. Not quite sure how I'd apply this. But the histogram of the z scores vs. the index scores definitely makes sense. Where does u come from? Or rather, how would I work with u across, say, 6 columns? $\endgroup$
    – Kyle Pennell
    Sep 10, 2020 at 20:47
  • $\begingroup$ If $U\sim\mathsf{Unif}(0,1)$ and $X$ has CDF $F_X,$ then $F_X^{-1}(U)$ is a random variable with CDF $F_X.$ $F_X^{-1},$ is called the quantile function of $X,$ so this is called the quantile function method of simulating realizations of $X.$ Accordingly, if $X$ has CDF $F_X,$ then $F_X(X)\sim\mathsf{Unif}(0,1).$ The usual symbol for the CDF of a std normal RV is $\Phi$ (pnorm in R; what is it in Python?). Thus, if $Z\sim\mathsf{Norm}(0,1),$ then $\Phi(Z)\sim\mathsf{Unif}(0,1).$ // In English: transforming a std normal by $\Phi$ gives a random variable $U=\Phi(Z)\sim\mathsf{Unif}(0,1).$ $\endgroup$
    – BruceET
    Sep 10, 2020 at 21:04
  • 2
    $\begingroup$ Computer packages such as R and Python can be enormously helpful in probability computations and statistical analysis, but it is a mistake to tie your fundamental understanding of concept of probability or statistics so directly to particular software that you can't discuss it outside that context. // Perhaps that's why this site generally discourages questions narrowly focused on debugging computer programs or implementing a kind of analysis specifically in one particular 'brand' of software: whether Excel, SPSS, Minitab, Stata, R, Python, MatLab, SAS, or other. $\endgroup$
    – BruceET
    Sep 10, 2020 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.