3
$\begingroup$

Let's say I am analysing an experimental design with treatment and control group which was based on random sampling design stratified on a number of covariates. When analysing the data I see two options:

  1. include the stratification variables as covariates
  2. set a complex survey design option that accounts for stratification

Option 1 is the option I see done in academic papers in my literature, but option 2 makes more sense to me. I am wondering whether to two procedures are equivalent or if option 2 gains more power?

$\endgroup$

1 Answer 1

1
$\begingroup$

Some thoughts

  • An important issue in this context is a correct estimation of standard errors, and therefore p-values, confidence intervals and thus overall levels of statistical significance of your independent variables. Being more specific, the accuracy of standard errors in complex sample designs is commonly compared against a Simple Random Sample (SRS). So compared to the SRS, standard errors under stratification design yield smaller standard errors. Now, since you have a Stratified Random Sample which implies a different probability of selection + stratification (in contrast to the SRS where the probability of selection is the same for all sampled units), I think it is generally difficult to speculate what standard errors will be like compared to the SRS.

  • However, I would think about the problem this way. Assuming stratification by some variables is indeed an important part of your design, calculating stratified standard errors will give you more confidence in the accuracy of your standard errors. Additionally, assuming that your stratified variables are for example gender or age group means that you have that data also recorded.

  • Now you may do the following as one way of going about this problem. You can indeed set a complex survey design with stratification. Then, if you have doubts about your stratification results, you can examine how your model is affected, if at all, by the addition of stratification covariates (really coded as dummy variables in this sense).

  • A cautionary note: assuming you also want to include a number of stratification variables as dummy covariates, please be aware that the inclusion of too many of such variables will make you capitalise on chance that some of the variables will be statistically significant by chance. For example, if you have 5 IVs and you decide to include 15 more dummy-coded covariates, this will total 20 IVs in your model, meaning that on average 1 of these variables is going to be statistically significant by chance. Hopefully, this might not apply to you, but it's good to be aware of this.

$\endgroup$
2
  • $\begingroup$ Thanks for your thoughts, it seems sensible to check both options. Perhaps, do you know literature on this subject? $\endgroup$
    – Papayapap
    Dec 19, 2019 at 15:17
  • $\begingroup$ Thank you for your comment. Yes, absolutely! I learned a lot from this specific book: Heeringa, S. G., West, B. T., & Berglund, P. A. (2010). Applied survey data analysis. London: CRC Press. If you can't get hold of it, please let me know. I may edit the answer to include a table that may be very useful for you. It shows how standard errors under different designs compare to the SRS $\endgroup$ Dec 19, 2019 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.