1
$\begingroup$

I have data indicating the number of counts per minute (so 60 rows in total - one for each minute - and # of events in that minute). I have ran the Shapiro - Wilk test which implies the data does not differ significantly from a Normal Distribution - can I therefore infer correctly that the counts are randomly distributed across each minute?

After spending some time researching I found a test called the "Runs test" which tests for randomness, however is normality enough?

$\endgroup$
1
  • 1
    $\begingroup$ Sort your dataset by count so that the counts increase over time. Give it to another researcher who doesn't know how you created it and ask them whether it looks random: what do you think they will say? Of course it doesn't look random to them. Now try to persuade them it is random by pointing out that the Shapiro-Wilk test (which doesn't care about the order of the data, and therefore will have the same result as before) doesn't detect any non-Normality. Do you think they will buy it? $\endgroup$
    – whuber
    Commented Jun 14, 2019 at 12:56

3 Answers 3

3
$\begingroup$

Your question is a bit confusing, but I'll try answering. First of all, if you assume that some random variable follows normal distribution, then by definition, it is "random". However your question seems rather to ask if the fact that something looks like normal distribution (on histogram, it passes a normality test etc.), does this imply that it is a result of random process. The answer to the latter question is: obviously not. You can easily write a computer program that deterministically produces results that are consistent with normal distribution, and by "deterministic" I mean here something even stronger then pseudo random generator: you can simply repeat each of the possible values the number of times that is proportional to the probability of observing it according to normal distribution.

grid <- seq(-6, 6, by=0.01)
prob <- dnorm(grid)
prob <- prob/sum(prob)
counts <- prob*1e6
x <- rep(grid, round(counts))

hist(x, breaks=100, freq=FALSE)
curve(dnorm(x), from=-6, to=6, lwd=2, col="red", add=TRUE)

enter image description here

$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your answer Tim. I suppose a follow question I would have is to ask how I can test if my data is random - specifically, that there is no bias to any particular minute in terms of the number of requests received in each of those minutes? $\endgroup$ Commented Jun 14, 2019 at 11:57
  • 1
    $\begingroup$ If you have a different question, post a new question. $\endgroup$
    – Glen_b
    Commented Jun 14, 2019 at 13:46
3
$\begingroup$

Normality is not enough for randomness, if by "randomness" you mean independent sampling (as indicated by mentioning the runs test.) A perfectly systematic, deterministic sequence can pass a test for normality, after all such a test is only looking at the marginal distribution.

An extreme example, in R:

library(tidyverse);library(snpar)
x <- ppoints(1000) %>% qnorm
x %>% shapiro.test
x %>% runs.test

> > 
    Shapiro-Wilk normality test

data:  .
W = 0.99995, p-value = 1

> 
    Approximate runs rest

data:  .
Runs = 2, p-value < 2.2e-16
alternative hypothesis: two.sided
$\endgroup$
0
$\begingroup$

Well, by "randomness", I assume you mean "independence" (i.e: what happens on minute $x$ has nothing to do with what happens on minute $y$) for any $x$ and $y$ you may choose. If that's the case, then:

First, normality does not imply independence. Since normality tests don't care about the order of the observations, you can have "normal" data that look like (-5, -3, -1, -.5, -0.25, 0.3, 0.45, 0.9, 2, 5), where we have an upward trend or even (-5.14, 5.18, -4.01, 4.21, -3.4, 3.1, -2.22, 2.11, -1.08, 0.97, -0.54, 0.76, -0.21, 0.32) where we have strong autocorrelations. My advice would be to perform some kind of independence test. For example, if you suspect there is order-based correlation, you can check that via ACF/PACF plots

Also, the reason why your data is not significantly different from normality is that you don't have a large enough sample. Every real-world based data will eventually fail the Shapiro-test if you have enough of it as every small deviation from normality will be detectable

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.