4
$\begingroup$

I’m trying to figure out if there’s a calculation that can be done to show you have enough data to draw conclusions on trends within the data set.

I have made an observation on a class of equipment that it seems to throw an error code on a specific day only, more often than the other days. The events are independent (i.e., when one unit throws the code the others don’t). This error code occurs infrequently – I’ve only observed it 18 times in total over several years. Here is the breakdown:

$$\begin{array}{} \mathrm{Monday} & 3 \\ \mathrm{Tuesday} & 2\\ \mathrm{Wednesday} & 1\\ \mathrm{Thursday} & 2\\ \mathrm{Friday} & 1\\ \mathrm{Saturday} & 2\\ \mathrm{Sunday} & 6\\ \end{array}$$

My hypothesis is that the error code is caused by a design flaw in the unit that causes an infrequent failure. However, at face value it appears there’s a strong correlation between Sunday and failures, despite there not being anything unique occurring on Sunday – it’s just like any other day as far as the equipment is concerned.

I think that we just haven’t observed this error enough times. Is there a calculation that can be done to indicate when you’ve collected enough data?

If this error occurred more times, I’d bet the occurences would even out.

$\endgroup$
  • 2
    $\begingroup$ Do I understand you correctly that you want to know whether the number of errors per day is really (i.e., in the "population" of all possible runs on every day) equal and whether this imbalance you see here is merely sampling variability? $\endgroup$ – Momo Jan 14 '15 at 15:03
  • $\begingroup$ Yes, that's what I'm asking. Is the number of error on Sunday really due to the fact that I don't have enough samples yet. Luck of the draw led to 6 on Sunday and less on the other days so far. $\endgroup$ – Steve Jan 14 '15 at 15:42
5
$\begingroup$

The test used will determine how to assess how much data are needed. However, standard tests, such as the $\chi^2$, would seem to be inferior or inappropriate, for two reasons:

  1. The alternative hypothesis is more specific than mere lack of independence: it focuses on a high count during one particular day.

  2. More importantly, the hypothesis was inspired by the data itself.

Let's examine these in turn and then draw conclusions.

Standard tests may lack power

For reference, here is a standard test of independence:

x <- c(3,2,1,2,1,2,6)                            # The data
chisq.test(x, simulate.p.value=TRUE, B=9999)

X-squared = 7.2941, df = NA, p-value = 0.3263

(The p-value of $0.33$ is computed via simulation because the $\chi^2$ approximation to the distribution of the test statistic begins breaking down with such small counts.)

If--before seeing the data--it had been hypothesized that weekends might provoke more errors, then it would be more powerful to compare the Saturday+Sunday total to the Monday-Friday total, rather than using the $\chi^2$ statistic. Although we can analyze this special test fully (and obtain analytical results), it's simplest and more flexible just to perform a quick simulation. (The following is R code for $100,000$ iterations; it takes under a second to execute.)

n.iter <- 1e5                                    # Number of iterations
set.seed(17)                                     # Start a reproducible simulation
n <- sum(x)                                      # Sum of all data
sim <- rmultinom(n.iter, n, rep(1, length(x)))   # The simulated data, in columns
x.satsun <- sum(x[6:7])                          # The test statistic
sim.satsun <- colSums(sim[6:7, ])                # The simulation distribution
cat(mean(c(sim.satsun >= x.satsun, 1)))          # Estimated p-value

0.08357916

The output, shown on the last line, is the p-value of this test. It is much smaller than the $\chi^2$ p-value previously computed. This result would be considered significant by anyone needing 90% confidence, whereas few people would consider the $\chi^2$ p-value significant. That's evidence of the greater power to detect a difference.

Greater power is important: it leads to much smaller sample sizes. But I won't develop this idea, due to the conclusions in the next section.

A data-generated hypothesis gives false confidence

It is a much more serious issue that the hypothesis was inspired by the data. What we really need to test is this:

If there were no association between events and day of the week, what are the chances that the analyst would nevertheless have observed an unusual pattern "at face value"?

Although this is not definitively answerable, because we have no way to model the analyst's thought process, we can still make progress by considering some realistic possibilities. To be honest about it, we must contemplate patterns other than the one that actually appeared. For instance, if there had been 8 events on Wednesday and no more than 3 on any other day, it's a good bet that such a pattern would have been noted (leading to a hypothesis that Wednesdays are somehow error-inducing).

Other patterns I believe likely to be noted by any observant, interested analyst would include all apparent clusters of data, including:

  • Any single day with a high count.

  • Any two adjacent days with a high count.

  • Any adjacent days with a high count.

"Adjacent" of course means in a circular sense: Sunday is adjacent to Monday even though those days are far apart in the data listing. Other patterns are possible, such as two separate days with high counts. I will not attempt an exhaustive list; these three patterns will suffice to make the point.

It is useful to evaluate the chance that a perfectly random dataset would have evoked notice in this sense. We can evaluate that chance by simulating many random datasets and counting any that look at least as unusual as the actual data on any of these criteria. Since we already have our simulation, the analysis is a matter of a few seconds' more work:

stat <- function(y) {
  y.2 <- c(y[-1], y[1]) + y         # Totals of adjacent days
  y.3 <- y.2 + c(y[-(1:2)], y[1:2]) # Totals of 3-day groups
  c(max(y), max(y.2), max(y.3))     # Largest values for 1, 2, 3 days
}
sim.stat <- apply(sim, 2, stat)
x.stat <- stat(x)
extreme <- colSums(sim.stat >= x.stat) >= 1
cat(p.value <- mean(c(extreme, 1)))

0.3889561

This result is a much more realistic assessment of the situation than we have seen before. It suggests there is almost no objective evidence that events are related to day of week.

Conclusions

The best solution, then, might be to conclude there likely is not anything unusual going on. Keep monitoring the events, but do not worry about how much time will be needed to produce "significant" results.

$\endgroup$
  • 1
    $\begingroup$ Nice discussion about data dredging. Kuiper's V fails to reject the null as well ($V = 1.68$, and using the exact distribution from my linked paper in the comments to Harvey's post). Part of the reason I wrote the paper was to prevent chasing the noise -- it is easy to look at that at those numbers and think there is a pattern, but does not reject the null of days being chosen with equal probability. It was intended to be a quick test given the data at hand -- something I found use for as a crime analyst and assessing patterns like this multiple times a month. $\endgroup$ – Andy W Jan 15 '15 at 19:11
  • 1
    $\begingroup$ @Andy Your paper addresses this question well--thank you for the link! It covers an issue I skipped over (for brevity and simplicity, not because I find it unimportant); namely, computing and using the exact distributions of the test statistics. $\endgroup$ – whuber Jan 15 '15 at 19:20
3
$\begingroup$

It seems to me that you want to test the null hypothesis that errors occur regardless of day of the week. To test this, you can perform a chi-square test that compares the actual observed number of errors on each day of the week with the expected number under the null hypothesis. That expected number for each day is the total number of errors observed (17) divided by 7.

$\endgroup$
  • 1
    $\begingroup$ Agree. Here I show how to generate the exact distribution under the null (since the asymptotic chi-square approximation is questionable given so few observations). In my paper it is a different application (crimes by a repeat offender on different days of the week) but I show that the exact test is reasonably powerful for particular alternatives. $\endgroup$ – Andy W Jan 14 '15 at 17:14
  • 1
    $\begingroup$ Here is the analysis for this example. The $\chi^2$ value is $7.3$, and under the exact null of all days being equally probable the p-value is $0.3$. So there is insufficient evidence in this example to suggest that errors do not occur with equal probability for each day of the week. $\endgroup$ – Andy W Jan 14 '15 at 17:19
  • $\begingroup$ @Andy, this deserves to be posted as a separate answer! $\endgroup$ – amoeba says Reinstate Monica Jan 15 '15 at 17:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.