1
$\begingroup$

I want to draw a cumulative probability distribution but I don't know the distribution of my data and I did normality test on my whole data(100,000 time difference data points) but it failed to show normal distribution. I don't know if I should use the whole data points to do normality test or not. Can I still draw cumulative probability distribution?

Looking at the example below, it does not generate random data points from a certain distribution but it can still draw cumulative probability distribution.

https://stackoverflow.com/questions/9378420/how-to-plot-cdf-in-matplotlib-in-python

$\endgroup$
2
  • 5
    $\begingroup$ If you have that many data points, why don't you simply plot the empirical CDF? You can smooth it with LOESS or spline regression in order to obtain a smooth curve instead of a step function. Another apporach would be to directly intergrate the kernel density estimate of your probability density: stats.stackexchange.com/a/296317/244807 $\endgroup$
    – cdalitz
    Oct 30, 2021 at 11:43
  • 4
    $\begingroup$ Good Q (+1). Roughly speaking, the ECDF comes close to the actual CDF, especially for large datasets. Because binning of histograms is arbitrary, it is often better to use ECDFs than to use histograms. If you like histograms, then plot the kernel density estimator (KDE) along with the histogram. // If you want to judge whether data are normal, a normal Q-Q plot may give more relevant information than formal goodness-of-fit tests (such as Kolmogorov-Smirnov or Shapiro-Wilk). Also tests may not be available for huge samples. // My Answ illustrates with some plots and some tests. $\endgroup$
    – BruceET
    Oct 31, 2021 at 2:39

1 Answer 1

4
$\begingroup$

In practice, a very large sample---supposedly from a normal population---can 'fail' a normality test either (a) because the population is not even close to normal or (b) the sample is from a very nearly normal population, but has some unimportant quirk that leads to 'failure' of standard normality tests. Many statistical computer programs (including R) will not perform Shapiro-Wilk normality tests for sample larger than a few thousand.

However, you can (a) make an empirical CDF (ECDF) of the data and compare it with the CDF of an appropriate actual normal CDF, (b) make a normal probability plot (normal quantile-quantile plot) to see if it is approximately linear, or (c) test randomly chosen, moderately large subsamples of the huge sample to see if they, 'pass' the test as normal.

Simulated precisely-normal sample. Consider the following sample x of size $n = 100\,000$ from $\mathsf{Norm}(\mu = 50, \sigma = 7).$

set.seed(2021)
x = rnorm(10^5, 50, 7)
summary(x);  length(x);  sd(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  18.38   45.30   50.01   50.03   54.77   83.19 
[1] 100000       # sample size
[1] 7.036199     # sample SD

Matching ECDF to CDF. In the plot below, the ECDF of x is shown along with the CDF (broken blue) of $\mathsf{Norm}(50, 7).$

hdr="ECDF (black) with NORM(50,7) CDF (blue)"
plot(ecdf(x), main=hdr)
curve(pnorm(x, 50, 7), add=T, lwd=3, lty="dashed", col="blue")

enter image description here

[If you do not know the mean $\mu$ and variance $\sigma,$ you could approximate $\mu$ as mean(x) and $\sigma$ as sd(x),$ which are excellent approximations for such a large sample, to get a very similar plot.]

Normal Q-Q plot. A normal probability plot of x has points that fall very close to a straight reference line (except possibly for a few points near the ends of the plot).

qqnorm(x)
qqline(x, col="red", lwd=2)

enter image description here

[This method works whether or not you know the population mean and standard deviation.]

Tests on subsamples of moderate size.

Kolmogorov-Smirnov test of match to particular normal distribution: In R, the procedure ks.test will test to see if a sample was taken at random from a particular normal population---with specified mean and SD. We show Kolmogorov-Smirnov tests for three sub-samples of size $1000.$ All three 'pass' the test; that is, the test fails to reject the null hypothesis that data are from $\mathsf{Norm}(50, 7).$

set.seed(1234)
ks.test(sample(x,1000), pnorm, 50, 7)$p.val
[1] 0.8436103
ks.test(sample(x,1000), pnorm, 50, 7)$p.val
[1] 0.7222755
ks.test(sample(x,1000), pnorm, 50, 7)$p.val
[1] 0.1864064

Shapiro-Wilk test to unspecified normal distribution. In R, the procedure shapiro.test will test a sample of size $n_1 \le 5000.$ Here are P-values of three samples of size $n_1=1000,$ all of which fail to reject the null hypothesis of normality at the 5% level. [The null hypothesis of a Shapiro-Wilk test is that data match some normal distribution---mean and SD not specified.]

set.seed(1030)
shapiro.test(sample(x, 1000))$p.val
[1] 0.5004599
shapiro.test(sample(x, 1000))$p.val
[1] 0.6135974
shapiro.test(sample(x, 1000))$p.val
[1] 0.9206584

Addendum. When data are not exactly normal. Suppose our data are heights of students (inches) whose heights come 50:50 from two distributions: $\mathsf{Norm}(65, 4)$ and $\mathsf{Norm}(69, 4).$ [In reality, distributions of almost all human populations are mixtures of several distributions.]

set.seed(2021)
x1 = rnorm(50000, 65, 4)
x2 = rnorm(50000, 69, 4)
y = sample(c(x1,x2))
summary(y)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  46.93   63.93   67.02   67.02   70.08   85.17 
length(y);  sd(y)
[1] 100000     # sample size
[1] 4.493029   # sample SD

For many practical purposes this mixture distribution is essentially $\mathsf{Norm}(67, 4.5).$ (a) The histogram of $100\,000$ students is well matched by its density estimator (dashed brown); both "look normal." (b) More precisely, the ECDF is similar to the CDF of $\mathsf{Norm}(67, 4.5).$ (c) Also, a normal Q-Q plot is essentially linear.

enter image description here

As is often the case, formal tests give a mixed message. Based on a sub-sample of $5000$ students, the K-S test (known for its poor power even for moderately large samples) does not reject the null hypothesis that the population is $\mathsf{Norm}(67, 4.5).$ However, the Shapiro-Wilk test rejects the null hypothesis that y1 is normal at all---regardless of parameters.

set.seed(123)
y1 = sample(y, 5000)

ks.test(y1, pnorm, 67, 4.5)$p.val
[1] 0.1272236

shapiro.test(y1)$p.val
[1] 0.01925218

[For what it's worth, the K-S test does reject the entire sample y of size $100\,000$ at the 2% level.]

Note: If you somehow know that the target distribution is normal and have a huge sample, then you may get a good result by estimating $\mu$ by $\bar X$ and $\sigma$ by the sample standard deviation $S,$ and using the CDF of $\mathsf{Norm}(\bar X, S).$ That may be about as good as using the ECDF of the sample.

Simulation showing estimation of $P(X \le 55)$ for $X \sim \mathsf{Norm}(50, 7)$ with samples of size $10\,000:$

set.seed(1776)
m = 10^5;  P.55 = p.55 = numeric(m)
for(i in 1:m) 
 {x = rnorm(10^4,50,7)
 P.55[i] = pnorm(55, mean(x),sd(x))
 p.55[i] = mean(x <= 55)
 }
mean(p.55)            # usimg ECDF
[1] 0.7624912
2*sd(p.55)/sqrt(m)    #  95% marg of sim err
[1] 2.693748e-05
mean(P.55)            # estimating mean & SD
[1] 0.7624899
2*sd(p.55)/sqrt(m)    #  95% marg of sim err
[1] 2.693748e-05
pnorm(55,50,7)        # exact
[1] 0.7624747
$\endgroup$
1
  • $\begingroup$ As far as you have revealed so far, Yes. I don't know how you would go about testing multiple samples as once. // If you were using variables in a t-test, ANOVA, or regression, then it is the "residuals from the model" that should be tested for normality.. $\endgroup$
    – BruceET
    Nov 10, 2021 at 1:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.