5
$\begingroup$

Background: I have conducted some testing on a random sample of n=20 parts. The data is variable and I know nothing about the population statistics. I would like to use the data from this sample to make statements about the population (in particular, I would like to say with 95% confidence that the 99th percentile of the population is above some value).

I have read about the importance of checking normality for this type of analysis but have also read good posts about "low power" of a normality test when the sample size is relatively small. Those posts recommend that you check "graphically" and also to transform the data as necessary even if a basic normality check indicates there isn't a compelling reason to reject the null hypothesis that the data is normal.

I transformed the data using a few basic transformations (square root, inverse, etc.) just to see what it would look like. None look drastically different but they do show different "p" values.

My question: Should I choose and use a transformation with the highest "p" value even if a (low powered) normality check of the untransformed data is not below .05?

As I understand it, that transformed data set would have the lowest probability of actually "not being normal". I've attached images of the normality checks I ran on the three sets (untransformed, transformed w/ square root, transformed w/ inversion) using Anderson-Darling.

The only stats tool I have available to me is Minitab.

***EDIT: The reason that I thought normality would be important is that it is typical in my industry to perform these types of reliability calculations using tables of "One-sided and Two-Sided Statistical Tolerance Limit Factors (k)" and the tables are only shown/valid for normal distributions. See, for example, "Tables for One-Sided Tolerance Limits" Industrial Quality Control, vol. XIV, no 10. You do this by calculating X +/- ks where X is sample mean, s is sample std dev, and k is from a table and is a function of desired confidence, reliability, and sample size.

enter image description here

enter image description here

enter image description here

$\endgroup$
6
  • $\begingroup$ Are your data discrete? If so, how could they be normal? $\endgroup$ Oct 7, 2013 at 19:14
  • $\begingroup$ You ask for a 99% coverage, 95% confidence lower tolerance limit based on just 20 observations. It therefore may be sensitive to your distributional assumptions, which are hard to assess. (Your data as they stand come closest to normality on a log scale, but if you were to remove (say) just a single one of those fours, it would be best not to re-express the data at all!) You might consider finding either a nonparametric tolerance limit or obtaining a tolerance limit for a very flexible, rich set of distributions, or studying how the TL varies with the underlying distributional assumption. $\endgroup$
    – whuber
    Oct 7, 2013 at 20:00
  • $\begingroup$ @ gung: the data are not discrete but my measuring device can only measure to the nearest 0.5 units. So maybe this is considered discrete? I could investigate getting a more sensitive measurement system but that would be resource intensive. @whuber thank you, I will try that. Note my edit in the original post explaining why I approached this in the way I did $\endgroup$
    – user31189
    Oct 7, 2013 at 20:56
  • $\begingroup$ Yes, that makes the data discrete, even though the underlying quantity you are measuring might not be. You can exploit this to increase the lower tolerance limit (LTL) from $5.5$ to $5.75$, because you have three values that are (presumed) definitely above $5.75$. This procedure is not quite as robust as when the underlying values are presumed discrete: if you were to lose the two highest values, then the LTL would drop to $5.25$. $\endgroup$
    – whuber
    Oct 7, 2013 at 21:30
  • 1
    $\begingroup$ "I have read about the importance of checking normality for this type of analysis" -- really? I'm curious to discover why this unnamed source feels is it important to check normality for a one-sided interval on a quantile, and further, why a hypothesis test in particular was thought to be of value for that purpose. (In any case, your data are quite plainly non-normal.) Do you recall where you saw this advice? $\endgroup$
    – Glen_b
    Oct 7, 2013 at 21:49

2 Answers 2

5
$\begingroup$

You can say a lot about a lower bound for an upper percentile, even with small amounts of data.

Suppose the 99th percentile of the true (but unknown) distribution from which $n$ values are obtained (independently) is the number $x_{0.99}$. Then the chance that $k$ or more of your data exceed $x_{0.99}$ is given by the Binomial distribution and equals

$$\sum_{j=k}^{n}\binom{n}{j}(0.01)^j(1-0.01)^{n-j}.$$

For instance, suppose the 99th percentile were less than $6$ but larger than $5.5$, which are the two largest values in the dataset (the $6$ appears three times). Then $k=3$; a quick computation shows that the chance of obtaining three or more values above the 99th percentile out of $n=20$ is a mere $43$ per million: that's really small and much smaller than the $100 - 95\%$ risk you are willing to run with $95\%$ confidence. Put another way, we can assert with as much as $100(1 - 43/1000000) = 99.9957\%$ confidence that the 99th percentile is no lower than $5.5$.

This calculation would not change under any monotonic, increasing re-expression of the data (such as the square root, logarithm, or negative reciprocal). Thus the question of data distribution is practically irrelevant.

This calculation is called a non-parametric lower tolerance limit for the upper $99$th percentile with $95\%$ confidence. There are parametric calculations that rely on knowledge (or assumptions) of the underlying distribution governing the data. However, the apparently discrete nature of these data (they occur in multiples of $1/2$) shows that any continuous distribution, such as a Normal, is at best a crude approximation. Thus it is safest--and scarcely gives up any precision--to use a method that assumes little or nothing about the distribution. That's what "nonparametric" means, and shows it is a robust procedure. For instance, if you were to lose as much as a quarter of these data, then even in the worst case (where the highest five values went missing) your conclusion would not change. (Your statements of the confidence level might vary, but they would always exceed the desired $95\%$.)

$\endgroup$
0
2
$\begingroup$

Any attempt to use the data to fit the distribution, when the choice is among 3 or more distributions, will result in at most tiny improvements over nonparametric methods, due to model uncertainty. For example if you try different distributions to get agreement with the empirical CDF the true variance of the final estimates will equal the variance of the empirical CDF.

$\endgroup$
2
  • $\begingroup$ This is true, but how are estimates of variance germane to estimating lower bounds for upper percentiles? (It might appear that they are, but it seems to me the connection is so tenuous as to be almost irrelevant.) $\endgroup$
    – whuber
    Oct 7, 2013 at 20:26
  • $\begingroup$ My comment applied to the part of the question about trying different transformations to achieve normality. The OP was trying different transformations to then be able to estimate quantiles. This is almost futile. $\endgroup$ Oct 7, 2013 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.