Significance factors

How can I determine a significance factor?

For example,

If 1/1 samples returns positive, the ratio is one. However, if 9/10 samples return positive, the ratio is .9, but is better because of the number of samples tested. Is there any way to statistically rank these test scenarios. I know it may be up to subjectivity, especially when deciding harder ones, such as

9/10 and 85/100, but any approach would be appreciated.

Based on the clarification in the comments, the number of mutations in a gene is binomially distributed and so you can use a formal statistical test to test for the difference between two population proportions.

Consider gene A and gene B with sample sizes $n_A$ and $n_B$, probability of each mutation $p_A$ and $p_B$ and let the proportion of mutations you observe in gene A and gene B be $\hat{p_A}$ and $\hat{p_{B}}$, respectively. You want to know whether gene A has a higher proportion of mutations than gene B. A statistical test at significance level $\alpha$ is the following:

$$H_0: p_A-p_B=0\quad\text{vs}\quad H_1:p_A-p_B\ne 0.$$

A test statistic is

$$T=\dfrac{\hat{p_A}-\hat{p_B}}{\sqrt{\hat{p}\left(1-\hat{p}\right)\left(\dfrac{1}{n_A}+\dfrac{1}{n_B}\right)}}\,\sim\mathcal{N}(0,1)\text{ under }H_0$$

where $$\hat{p}=\dfrac{\hat{p_A}+\hat{p_B}}{n_A+n_B}.$$

A $p$-value for this test gives you the statistical ranking that you're looking for as it takes sample size into consideration.

Obviously, having 300 genes makes your problem more difficult to approach in this manner: comparing every gene with every other gene would give you 44850 tests! To avoid as many tests as possible, I would recommend starting off with a gene that has a very high proportion and high sample size. Then, computing the $p$-values for the 299 tests, you can forget all the genes that have a lower proportion than gene C except for the top 10 (those with the ten highest $p$-values).

• I think you are mistaken in the last paragraph. If I compare a gene with every other gene, it won't be 300!. It should be 300 c 2, or 44,850 tests, right? @MLaz Aug 4, 2014 at 0:00
• But wow man, I learned this in my AP Stats class last year (in junior year)...kind of disappointed I wasn't able to recall a 2-sample z test for proportions by myself. Just to clarify, once I get the test statistic, I can get p by finding z > T? Aug 4, 2014 at 0:19
• If you compare gene A with all the others that's 300 tests. Then compare gene B with all but gene A and that's 299 tests, and so on to 300! tests altogether, I believe. For the two-tailed test I described, you need the probability in both tails, $P(Z>|T|)$. You could also consider a one-tailed test, and then it would be $P(Z>T).$
– MLaz
Aug 4, 2014 at 0:42
• No you have a misunderstanding of factorial. It is 299 + 298 ... 1, or (299 * 300)/2. If you have R, just do sum(1:299). Factorial is 300 x 299 x 298. Completely exponential. Okay, and the second part seems okay. Let us say, that I am able to generate a test result for each pair of genes. (44,850 tests) - I can do it, don't worry about how I will. What then? How do I rank them from there? Aug 4, 2014 at 0:46
• So, I think I know how. I calculate the test statistic for each combination of genes, and find the highest test statistic. I link it back to the genes, let's call them gene E and gene F, for argument's sake. If the highest test statistic is produced by E and F, I can conclude E is the gene with the highest proportion-samplesize stats, and F is the lowest. Then look for the combinations of E with other genes that result in the lowest test statistic. The lower the test statistic, the closer it is, thus hereby ranking the genes. Aug 4, 2014 at 1:20

I'm presuming you're doing a hypothesis test to check the significance of a covariate. In doing a hypothesis test, you would $\textit{always}$ use all the samples in one test, not do a test for each sample separately.

As you correctly point out, more samples is better. In a hypothesis test there are two errors: a type 1 error, $\alpha$, which is set before performing the hypothesis test (often set to be 5%). The other error you can make is a type 2 error, $\beta$, which is dependent on the test you are using, among other things. Using more samples, the probability of a type 2 error will decrease.

In terms of a ranking, the power ($=1-\beta$) could be used to rank a test. Clearly maximising the power means the smallest type 2 error. The power is normally considered when deciding between which hypothesis test to use. Tests are called uniformly most powerful if they give the best power and a test that is uniformly most powerful would be preferred over any other test.

It is important not to decide on the test after you have performed it, as this will bias your conclusions. You would, beforehand, decide on the acceptable probability of a type 1 error and a type 2 error and then collect your samples; you must not choose the number of samples after you've performed a test.

• Hmm, I don't think that's what I am doing. Each "scenario" is different. For example, I am looking at about 300 genes, which I need to rank by the frequency with which they are mutated in samples. Gene A is let's say mutated 9/10 times, while Gene B is mutation 99/100 times. Thus, Gene B is ranked higher in terms of mutations that Gene A. Would this still apply? @MLaz Aug 3, 2014 at 20:15
• Okay, I see. Can you assume, for each gene, that each mutation occurs independently and with equal probability of mutating (within that gene)?
– MLaz
Aug 3, 2014 at 20:23
• Just to clarify. I take 100 samples of let's say a certain disease. In this case, I look for mutations in Gene B in all 100 samples, and find that 99 of these samples have mutation in Gene B. But for Gene A, I test only 10 samples. And find that 9 have mutations. You may ask, why test the genes different amounts? This is usually due to funding/how easy it is to test/prior studies. Anyways, since Gene A is mutated in 9/10 samples and Gene B in 99/100, I can conclude Gene B. To answer your question, yes I can assume that @MLaz Aug 3, 2014 at 20:25
• Okay, so the number of mutations in a gene follows a binomial distribution. Are you then interested in the probability of a mutation, p? Or are you interested in how significantly non-zero the probability of p is?
– MLaz
Aug 3, 2014 at 20:36
• It isn't number of mutations in a gene I am looking for. Let's say I have 100 DIFFERENT patients. I test Gene B in all 100 patients, and it comes out that 99 of the patients have 1+ mutations in the Gene B. So it isn't number of mutations. I am talking about how likely a patient will have a mutation in a certain gene (based on past results - like empirical) So If I was to get a patient, it is 99% likely they will have a mutation in Gene B, based on 100 samples. Aug 3, 2014 at 21:02