Timeline for Reduced effective sample size in balanced population after inverse probability treatment weighting
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 26, 2023 at 18:40 | comment | added | Noah |
@arielhasidim P-values don't consider sample sizes. They come from a formula computed from the estimate and its standard error. The standard error is computed correctly incorporating the weights if a robust SE is used as recommended in the WeightIt vignette. Please read the document because it explains the various modes of estimating SEs and CIs and their advantages and disadvantages.
|
|
| Jun 26, 2023 at 14:41 | comment | added | arielhasidim | @Noah Thanks again. And the last question? ("is the p-value from marginaleffects::avg_comparisons considering the ESS or the actual sample size?") | |
| Jun 26, 2023 at 14:29 | comment | added | Noah |
@arielhasidim There is no rule of thumb except the same rules you woudl apply to a regular sample size. That means you should od a powr analysis before the weighting to see the smallest ESS you can have to detect the effect size of interest. I would guess 100 units in each group would be too small to detect many meaningful effect sizes. I explain in the WeightIt vignette on estimating effects how to validly estimate standard errors and confidence intervals (and p-values) after weighting.
|
|
| Jun 26, 2023 at 9:18 | comment | added | arielhasidim | Thank you @Noah! I'm also pondering the consequences of significantly reduced ESS due to significant weighting, and I'm worried about the limit. Let's say I have 100 treated cases and 1K controls, and I use full matching to estimate ATT. Is there a rule of thumb about the ESS? Should I aspire not to go under a 1:1 ratio with the treated? And maybe more importantly, is the p-value from marginaleffects::avg_comparisons considering the ESS or the actual sample size? | |
| Apr 6, 2023 at 15:01 | vote | accept | user19745561 | ||
| Apr 6, 2023 at 15:01 | comment | added | user19745561 | Thank you so much! I will mark as answered and upvote! Thank you again | |
| Apr 6, 2023 at 14:17 | comment | added | Noah |
Stabilizing weights doesn't really do anything for a single time point treatment. It doesn't change balance or the ESS. It can slightly change the effect estimate if you use a certain kind of model. You can use trim() and just use summary() or bal.tab() to view the ESS after trimming. No units are dropped when trimming, so the sample size is the same. If you want to drop units from the sample, you can use estimand = "ATOS" or drop them manually, but I don't recommend this. Use a weighting method that yields good properties off the bat instead of trimming or dropping observations.
|
|
| Apr 5, 2023 at 21:46 | comment | added | user19745561 | Thank you! You have been incredibly clear. I have a question: as far as I understand in order to avoid extreme weights you can either stabilize the weights (stabilize=TRUE in WeightIt) or you can perform "trimming" with trim(). However, trimming from what I understand can mean either removing the observations responsible for the extreme weights or - as in trim() - "setting all weights higher than that at a given quantile..." If I choose to "trim" by deleting observations, how can I obtain the trimmed sample size of my population? Is it better in your opinion to use trim() or stabilize weights? | |
| Apr 5, 2023 at 16:00 | history | answered | Noah | CC BY-SA 4.0 |