0
$\begingroup$

I had a non-normal distribution of my variable of interest which required log10-transformation to reduce outliers. This was then standardized, to a mean of 0 and sd of 1 (z-score?). I now want to obtain percentile values from the standardized values. For example, how do I obtain the 75th percentile of my variable given I know it was not normal to begin with and even after log10-transformation it is not normally distributed. Thanks for insight.

$\endgroup$
  • 1
    $\begingroup$ So you have large number of observations of the said distribution? Sample percentile won't work for you? $\endgroup$ – Art Sep 5 at 4:16
  • $\begingroup$ ~450 observations. Can you expand on sample percentile? In it's standardized and transformed version, getting the Xth percentile is straightforward? $\endgroup$ – mindhabits Sep 5 at 5:05
  • $\begingroup$ 75th percentile = upper quartile. $\endgroup$ – Nick Cox Sep 5 at 5:10
  • 2
    $\begingroup$ I think so since all the transformations you mentioned preserve rankings $\endgroup$ – Art Sep 5 at 5:13
2
$\begingroup$

You may have started with data somewhat similar to the 450 observations in the vector w, which are summarized below using R:

summary(y);  sd(y)
Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
 914    27299    83975   380413   220675 16507340 
[1] 1381348  # sd

enter image description here

The transformation $W = \log_{10}(Y)$ gets rid of some of the skewness, and considerably reduces the number of boxplot outliers.

 w = log10(y);  summary(w);  sd(w)
  Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 2.961   4.436   4.924   4.930   5.344   7.218 
[1] 0.7004566  # sd

enter image description here

Finally, we transform to get a random variable $Z$ with $E(Z) = 0$ and $SD(Z) = 1.$ That is. $Z = \frac{W - \bar W}{S_W}.$ [The histogram shows that $Z$ is not normal.]

z = (w - mean(w))/sd(w)
summary(z);  sd(z)
      Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
 -2.810906 -0.705027 -0.008336  0.000000  0.590705  3.265995 
 [1] 1  # sd

enter image description here

Notice that the the summaries have given sample upper quartiles (75th percentiles throughout: 220675 for $Y,$ 5.344 for $W,$ and 0.590705 for $Z.$ Because the transformations have not changed the order of the observations, the sample quartiles are related according to the transformations: $\log_{10}(220675) = 5.344$ and $(5.344 - 4.930)/0.700 = 0.591.$

 log10(220675)
 [1] 5.343753
 (5.343753 - mean(w))/sd(w)
 [1] 0.5907055

With a sample as large as $n = 450,$ sample quartiles should not be very far from population quartiles. Because of the way I simulated the data, I know that the exact 75th quantile of the distribution of the $W$'s is 5.457, which is not far from the upper quartile 5.344 of the observed values of $W.$ Of course, a larger sample size would tend to give a better match between sample and population upper quartiles.

qgamma(.75, 50, 10)
[1]  5.457062

One style of 95% nonparametric bootstrap confidence interval for the population 75th percentile is $(5.27, 5.41).$ Unfortunately, our sample of 450 observations was an 'unlucky' one, with an unusually low 75th percentile, so this CI does not quite include 5.457. [I could have started over with a different seed to get a 'more cooperative' sample of 450, but it is wrong to give the impression that 95% CIs are 100% CIs.]

set.seed(2019)
d.re = replicate(2000, quantile(sample(w, repl=T),.75))
quantile(d.re, c(.025, .975))
    2.5%    97.5% 
5.267784 5.413533 
$\endgroup$
2
$\begingroup$

Let's say you have an RV $X$. Your transformations are all included in this function which result in another RV $Y = f(X)$. My understanding is that you'd like to know what the 75th percentile of $Y$ is. If that's the case, you could just use the samples (450 observations of $\hat Y$) to get an estimate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.