Normality Testing - Choose the transformation that makes the data "most normal"?

Question

Background: I have conducted some testing on a random sample of n=20 parts. The data is variable and I know nothing about the population statistics. I would like to use the data from this sample to make statements about the population (in particular, I would like to say with 95% confidence that the 99th percentile of the population is above some value).

I have read about the importance of checking normality for this type of analysis but have also read good posts about "low power" of a normality test when the sample size is relatively small. Those posts recommend that you check "graphically" and also to transform the data as necessary even if a basic normality check indicates there isn't a compelling reason to reject the null hypothesis that the data is normal.

I transformed the data using a few basic transformations (square root, inverse, etc.) just to see what it would look like. None look drastically different but they do show different "p" values.

My question: Should I choose and use a transformation with the highest "p" value even if a (low powered) normality check of the untransformed data is not below .05?

As I understand it, that transformed data set would have the lowest probability of actually "not being normal". I've attached images of the normality checks I ran on the three sets (untransformed, transformed w/ square root, transformed w/ inversion) using Anderson-Darling.

The only stats tool I have available to me is Minitab.

***EDIT: The reason that I thought normality would be important is that it is typical in my industry to perform these types of reliability calculations using tables of "One-sided and Two-Sided Statistical Tolerance Limit Factors (k)" and the tables are only shown/valid for normal distributions. See, for example, "Tables for One-Sided Tolerance Limits" Industrial Quality Control, vol. XIV, no 10. You do this by calculating X +/- ks where X is sample mean, s is sample std dev, and k is from a table and is a function of desired confidence, reliability, and sample size.

Answer 1

You can say a lot about a lower bound for an upper percentile, even with small amounts of data.

Suppose the 99th percentile of the true (but unknown) distribution from which $n$ values are obtained (independently) is the number $x_{0.99}$. Then the chance that $k$ or more of your data exceed $x_{0.99}$ is given by the Binomial distribution and equals

$$\sum_{j=k}^{n}\binom{n}{j}(0.01)^j(1-0.01)^{n-j}.$$

For instance, suppose the 99th percentile were less than $6$ but larger than $5.5$, which are the two largest values in the dataset (the $6$ appears three times). Then $k=3$; a quick computation shows that the chance of obtaining three or more values above the 99th percentile out of $n=20$ is a mere $43$ per million: that's really small and much smaller than the $100 - 95\%$ risk you are willing to run with $95\%$ confidence. Put another way, we can assert with as much as $100(1 - 43/1000000) = 99.9957\%$ confidence that the 99th percentile is no lower than $5.5$.

This calculation would not change under any monotonic, increasing re-expression of the data (such as the square root, logarithm, or negative reciprocal). Thus the question of data distribution is practically irrelevant.

This calculation is called a non-parametric lower tolerance limit for the upper $99$th percentile with $95\%$ confidence. There are parametric calculations that rely on knowledge (or assumptions) of the underlying distribution governing the data. However, the apparently discrete nature of these data (they occur in multiples of $1/2$) shows that any continuous distribution, such as a Normal, is at best a crude approximation. Thus it is safest--and scarcely gives up any precision--to use a method that assumes little or nothing about the distribution. That's what "nonparametric" means, and shows it is a robust procedure. For instance, if you were to lose as much as a quarter of these data, then even in the worst case (where the highest five values went missing) your conclusion would not change. (Your statements of the confidence level might vary, but they would always exceed the desired $95\%$.)

Answer 2

Any attempt to use the data to fit the distribution, when the choice is among 3 or more distributions, will result in at most tiny improvements over nonparametric methods, due to model uncertainty. For example if you try different distributions to get agreement with the empirical CDF the true variance of the final estimates will equal the variance of the empirical CDF.