6
$\begingroup$

I would like to test a hypothesis that a database contains data that are correct in 95% or more of the samples.

I found this Hypothesis test for a proportion experiment design that looked appropriate.

I will randomly sample one-tenth of the population and mark each datum as correct or incorrect.

Here is a summary of the design:

  1. State the hypotheses: (Null Hypothesis: P >= 0.95, Alternative: P < 0.95)
  2. Formulate an analysis plan (Significance level is 0.05)
  3. Analyze sample data (calculate the σ and compute the z-score test statistic (z))
  4. Interpret results (Reject the null hypothesis iff the P-value is less than the significance level (0.05).)

I have a handful of questions:

  1. Is this an appropriate test to use if there is no magnitude of error (only that the datum is correct or not)?
  2. The StatTrek page says to stop after 10 successes and 10 failures, but to sample no more than 10% of the population. Are 10, 10, and 10 just helpful rules of thumb, or is there some other guiding principle?
Improve this question
$\endgroup$
6
  • $\begingroup$ If your goal is to provide evidence that the true proportion is greater than 95% then you probably want to switch your null and alternative hypothesis around. Typically you put what you want to show in your alternative hypothesis. $\endgroup$
    – Dason
    Dec 31, 2010 at 16:10
  • $\begingroup$ @Dason, Good point. One of my profs said that you draw a stronger conclusion from rejecting the null hypothesis. $\endgroup$
    – rajah9
    Dec 31, 2010 at 16:16
  • 3
    $\begingroup$ "Sample 10% of the population" has almost no basis in statistical theory or practical application. (In fact, it is contrary to statistical theory, which shows that usually the absolute sample size matters, not its percentage of the population.) More importantly, any such rule is assuming something about the trade-off between the costs of sampling and the value of improved decision making to you, in your particular circumstance. That's exactly the sort of thing you don't want a stats cookbook or glib Web site to decide for you! $\endgroup$
    – whuber ♦
    Dec 31, 2010 at 16:48
  • 2
    $\begingroup$ A good guiding principle is Wald's sequential testing theory. The Wikipedia article includes a (brief) example identical to your question (but testing for a 99% proportion rather than a 95% proportion): en.wikipedia.org/wiki/Sequential_probability_ratio_test $\endgroup$
    – whuber ♦
    Dec 31, 2010 at 16:54
  • 1
    $\begingroup$ @rajah9 That site looks balanced and reasonable. I did not read it in detail, but I looked for unsupported assertions (like "always do this" or "10 is a minimum", etc.) and didn't find any. I did find a straightforward applet and lots of references, both of which are good signs of quality. As always, don't rely on anybody's software until you have checked it out with data where you know the answer. $\endgroup$
    – whuber ♦
    Jan 4, 2011 at 20:33

1 Answer 1

Reset to default
2
$\begingroup$

Thank you, whuber, for making me aware of Wald's Sequential Probability Ratio Test (SPRT). At your recommendation, I will relist this Quantitative Skills site. They will give you an out-of-the-box table to determine whether to continue or stop testing.

I also took the time to research that site's references, and was directed toward a comprehensive article that is intended for medical testing, but is easily transferable to other domains. It is Increasing Efficiency in Evaluation Research: The Use of Sequential Analysis (Howe, Holly L., American Journal of Public Health July 1982, Vol. 72, No. 7, pp. 690-697.) This article may be downloaded in its entirety.

Since I have not seen SPRT in my stats courses, I will provide a cookbook that I hope will he helpful for the stackexchange community.

For my null hypothesis, I tested for a level of 95% correct. If, however, the level was below 80%, it would be a cause for concern. So I have

$p_1 = .95$ (null hypothesis), and $p_2 = .80$ (alternative hypothesis)

I will use $\alpha = 0.05$ and $\beta = 0.10$.

Howe shows a graph with two parallel lines, with plots of the cumulative errors. Testing continues while the cumulative errors (and in my case, cumulative count of correct data points) lie between the two lines.

If the cumulative errors exceed either line, then either:

  • accept the null hypothesis (if cumulative error count falls below the bottom line, $d_1$), or
  • reject the null hypothesis (if cumulative error count exceeds the top line, $d_2$).

Here are the equations. I am adding a denominator because it is used several times.

$denom = log\left [ \left ( \frac{p_2}{p_1}\right )(\frac{1 - p_1}{1 - p_2}) \right ]$

The slopes of the lines are the same, and represented by s.

$s = \frac{log\left ( \frac{1 - p_1}{1 - p_2} \right )}{denom}$

The intercepts, $h_1$ and $h_2$, are computed as follows:

$h_1 = \frac{log\left ( \frac{1 - \alpha }{\beta }\right )}{denom}$

$h_2 = \frac{log\left ( \frac{1 - \beta }{\alpha }\right )}{denom}$

I set up a spreadsheet with data point N going from 1 to 50. Then I added two columns for acceptance threshold ($d_1$) and rejection threshold ($d_2$).

$d_1 = -h_1 + sN$

$d_2 = h_2 + sN$

In my experiment,

$denom = -0.67669$

$h_1 = -1.44485$

$h_2 = -1.85501$

The values of $d_1$ at N=2, N=5, N=10 are 3.224, 5.893, 10.342.

I then added columns for success and cumSuccess. I picked data points until the cumulative number exceeded the acceptance threshold, and I accepted the null hypothesis.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.