3
$\begingroup$

I want to compare two proportions, say $p_1$ and $p_2$. I want to test the null that $\pi_1 = \pi_2$ where $\pi_i$ is the true unknown proportion for sample $i$. The problem is that I do not know the sample size which underlies the calculation of $p_2$ and hence cannot use the standard two-sample, independent proportion test.

Can I, instead, assume that $p_2$ is a fixed value and then perform a standard one-sample, proportion test? Is there a better approach?

$\endgroup$
  • $\begingroup$ So, for $p_2$ someone just handed you a number, is that correct? What do you have for $p_1$, a vector of $1$s & $0$s? $\endgroup$ – gung Oct 6 '13 at 3:15
  • $\begingroup$ @gung It is a bit complex. $p_2$ is the overall average of several studies each one of which has its own sample size. All I know is that the sample size of each one of these individual studies is probably very high (maybe in the range of a 1000 to, not sure about this). $p_1$ is a vector of $1$s and $0$s. $\endgroup$ – prop Oct 6 '13 at 3:29
  • $\begingroup$ Do you know the component proportions (ie, from the individual studies) that went into $p_2$? Most likely, $p_2$ should be a weighted average of those proportions, where the individual study sample sizes are the weights, not the simple average. Note that this fact adds additional complexity to @Glen_b's otherwise good answer below. $\endgroup$ – gung Oct 6 '13 at 3:31
  • $\begingroup$ Perhaps, it is weighted avg. I do not have access to that information. And, no I do not know the component proportions. $\endgroup$ – prop Oct 6 '13 at 3:33
  • 1
    $\begingroup$ @gung for a proportions test to make sense any time, you have to assume homogeneity; if that's not reasonable, then even if you can justify the comparison (what's the population about which your inference is being made?), you have bigger issues - like computing a bound on your standard error. $\endgroup$ – Glen_b Oct 6 '13 at 3:48
4
$\begingroup$

The uncertainty in $p_2$ may be very large, so no.

Imagine $p_2 = 0.67$.

Now that could be 67 out of 100, in which case the standard error of that estimate of $\pi_2$ is about 0.047, so $\pi_2$ might be 0.6 or 0.75 ... but probably won't be 0.5 or 0.95.

However, that $p_2$ might also be 2 out of 3, rounded off - and in the latter case, a very wide range of population proportions could reasonably have given 2 successes out of 3. Assuming $p_2$ is exactly $\pi_2$ would be silly.

But all is not lost.

There's another possibility.

Depending on its value it might be possible to put a lower bound on the $n$ and make progress that way. For example, if $p_2 = 0.412$, the smallest $n$ consistent with that is 17 (7/17 = 0.4117, rounded to 0.412) - no smaller sample size can produce 0.412 with rounding (whether off, up or down).

If you assume the $n$ is at least as large as the smallest $n$ that could have produced your sample proportion, then you will get an upper bound on the p-value. If you're lucky the smallest $n$ consistent with your proportion won't be something like 5.

The more significant figures you have for $p_2$, the better your chances of a useable lower bound on $n$. But if it's a value like 0.67... well, you better hope $p_1$ is a loong way away from it.

$\endgroup$
  • $\begingroup$ Actually, the sample size is probably very high (maybe, in the thousands) which would justify taking $p_2$ as a fixed value, no? $\endgroup$ – prop Oct 6 '13 at 3:31
  • 1
    $\begingroup$ It's not me you have to convince that it's justifiable. Do you have some way of putting a lower limit on how big it might be? Some real evidence that it's large? If you just say "it's probably really big", a journal editor or a referee are both likely to check whether the calendar says April 1. Who do you need to convince and what do you have to convince them with? How certain can you be that it isn't small? $\endgroup$ – Glen_b Oct 6 '13 at 3:32
  • $\begingroup$ Well, first I need to convince myself that I am not doing something shaky and your answer helps in understanding the role sample size plays. Unfortunately, no precise lower limit is available. $\endgroup$ – prop Oct 6 '13 at 3:38
  • $\begingroup$ If you can convincingly argue "it's at least 100", then you can simply take $n=100$; unless your $p_2$ is quite close to 0 or 1 and your $p$-value is close to your significance level, if you reject at some $n$, you'll reject at greater $n$. $\endgroup$ – Glen_b Oct 6 '13 at 3:39
  • $\begingroup$ If you can't make such an argument, the discussion that it's probably quite large (and what that might mean for a test) would come after you discuss what you can confidently show. $\endgroup$ – Glen_b Oct 6 '13 at 3:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.