1
$\begingroup$

Let's say we want to conduct an experiment with binary treatment.

How do I decide how many units do I need to assing the treatment to, to get internally valid results? I understand that you need to reach a certain number of units for detecting statistically significant effect. But in this case I am concern about finding the (average) causal effect of my treatment free of possible confounders. (In this case I don't care about external validity and generalization of my results to the broader population). I just want to be sure that I have enough units to control the confounders with randomization.

Thank you for answers.

Edit: I am basically trying to measure SATE and I am not sure what role does the statistical significance test play in this context. Can it tell me anything about the internal validity of the experiment, or is it strictily directed toward external validity and generalization to the broader population?

$\endgroup$
4
  • 1
    $\begingroup$ Doing proper sample size calculation is only possible with detailed information on study design, endpoints, hypotheses etc., along with significance level, power, intended analyses and ways to deal with multiple testing. You have not provided any of these and so I'd refer to the following little piece of art: youtube.com/watch?v=PbODigCZqL8 $\endgroup$
    – Michael M
    Jul 17, 2020 at 21:13
  • $\begingroup$ Oh, I see. I am not a scientist nor a student. I'm asking just out of my own curiosity. But do I understand correctly that the significance level is relevant even for internal validity? $\endgroup$
    – Nothingman
    Jul 18, 2020 at 6:26
  • $\begingroup$ Not necessarily a significance level, but something similar. In your case it could be the desired precision of the estimated validity measure, but this is (admittedly) pure speculation. $\endgroup$
    – Michael M
    Jul 18, 2020 at 7:28
  • $\begingroup$ I've edited the main post: I am basically trying to measure SATE and I am not sure what role does the statistical significance test play in this context. Can it tell me anything about the internal validity of the experiment, or is it strictily directed toward external validity and generalization to the broader population? $\endgroup$
    – Nothingman
    Jul 18, 2020 at 7:31

1 Answer 1

1
$\begingroup$

I would love to give you a simple and straight-forward answer, but the problem is: It depends (upon the variability through your confounders). If you conduct your experimental design in a way that you have conditionally (on the confounders) randomized experiments, these subsets/strata necessitate a sufficient size to be statistically significant. The answer for your question depends now on the stratum specific variability.

I would recommend you the Wikipedia Article on Stratified Sampling as a first overview: https://en.wikipedia.org/wiki/Stratified_sampling

Besides, these three sources are likely to be helpful:
https://www.kidney-international.org/article/S0085-2538(15)52974-8/fulltext
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4017459/
https://oem.bmj.com/content/62/7/500

$\endgroup$
2
  • $\begingroup$ Thanks. I went through the papers, but I think the main point of my question stand still. I understand that randomization of a treatment is one of the mechanism through which you reduce the effect of confounders, so the estimation of the causal effect of the treatment is easier. But how do you decide/measure that the randomization indeed worked and both of you groups (treatment and control) are actually same in average of observable and un-observable variables? Do I still need to care about statistical significance if my intention is to estimate causal effect of the sample? $\endgroup$
    – Nothingman
    Jul 18, 2020 at 7:11
  • $\begingroup$ Maybe more shorter reply: Can I measure the variability of confounders and/or decide wheter the randomization has sufficient effect? $\endgroup$
    – Nothingman
    Jul 18, 2020 at 7:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.