1
$\begingroup$

I'm researching the role of sender reputation in remediating Android vulnerabilities using large-scale notification campaign. There are 2 senders with different reputations and 1 as a control (no notification sent). I am measuring the effect of these notifications by the senders and finding out how many vulnerabilities were fixed when they were notified. I'm having a problem in the study design as I can't seem to figure out the sample size needed for a meaningful analysis. This is the contingency table so far.

          Fixed   Not fixed  Total
Sender1                        21
Sender2                        21
Control                        22

Total                          64

I know I have to do it pairwise and then use Multitest corrections (I'll be using Holm-Bonferrroni for this), but the question is whether the analysis will be meaningful or not? For that I have to calculate sample size needed. How to do that?

(The question might be very basic but I don't have a background in stats so pls help)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I will try to answer you question, please feel free to correct me if I understand you wrongly.

First, here is a sample size calculator for fishers exact test or use this package in R, and here an article. If you are interested in further explanation, I would suggest to google, fishers exact test is well described.

After this is said, to calculate sample size, you need prior knowledge. Sample size should NEVER be calculated based on the observations used for the actual analysis.

An alterative for using fishers exact test for calculating sample size is to use proportions.

Second you ask about correction for multiple testing. Unless you are looking at 1000 or more tests, I would never recommend correcting for multiple testing. Why? Because you simply increase type 2 errors when trying to decrease type 1 errors, and how can you make sure you do not want to perform a test later on the same data set? If you do so and are determined on correction you previous results are no longer valid but should be corrected for the new additional tests. Here is an article that comment on these problems.

I hope this may be of help

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hmmm ... that article is rather old and has been discussed at length. In any event, are you recommending that one take the smallest of a collection of p-values, and if less than 0.05, then one can claim that the result is real and repeatable? $\endgroup$
    – BigBendRegion
    Jun 20, 2021 at 22:19
  • $\begingroup$ I am not sure what you are asking. When you calculate sample size, you use a formula (an approximation or numerical solution) based on prior knowledge/expectation on the distribution of your variables. You calculate sample size to estimate how small a "change/difference" you can detect under your assumptions at with a specific power and significance level. It is related to how much information you need to say estimate something with a predefined certainty. You cannot conclude that anything is real or repeatable - with applied statistics nothing is certain, only certain to a degree. $\endgroup$
    – Kirsten
    Jun 21, 2021 at 9:08
  • $\begingroup$ It wasn't a comment about sample size, it was a comment about "I would never recommend correcting for multiple comparisons." Not correcting for multiple comparisons clearly runs a risk of making claims that are not repeatable. See the ASA statement on p-values, eg. $\endgroup$
    – BigBendRegion
    Jun 21, 2021 at 15:08
  • $\begingroup$ I am aware, that is a problem. However correcting for multiple testing runs the same risk. In multiple comparison correction: decrease the chance of observing a "significant" relation that isn't really "significant", and at the same time increasing the risk of not observing something "significant" that is actually "significant". That is why I would recommend to use your knowledge (subject specific + statistical) in interpretation the result instead of making a correction that gives the reader less information. Further correcting p-values is a simple procedure, the reader can easily do themself $\endgroup$
    – Kirsten
    Jun 28, 2021 at 7:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.