Timeline for Bootstrap-based robustness of a test checking. Is resampling from sample distribution a good procedure?
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 17, 2022 at 9:00 | comment | added | Tomás Navarro | Thank you for your interest. My aim is checking if t-test aplied to these particular samples from potentially unknown distributions (as you pointed out) is robust enough to be applied after to the original samples (no 0 centered). The question could be Can I trust t-test result when applied to these samples? Due to original population distributions are unknown the main assumption here (big one) is that sample distributions emulate population ones, as an alternative to "infer" the expected population distribution from the nature of the variable studied or previous information. | |
| May 17, 2022 at 0:22 | comment | added | awhug | Thanks that helps for context. In the linked answer, the data generating process was known, so it was easy to verify theoretical vs. empirical false positive rates of the t-test through simulation under a variety of known sampling distributions. Given you've only presented one dataset with two groups drawn from potentially unknown/unknowable distributions, it's still a bit unclear what you're after. Are you ultimately aiming to make some conclusion about the t-test itself, or the results of a t-test as applied to a particular sample? What do you propose your analysis tells you? | |
| May 16, 2022 at 10:07 | comment | added | Tomás Navarro | Here the top answer explains a Monte Carlo method to check robustness: stats.stackexchange.com/questions/242037/… I propose instead to use the empirical distribution of the input samples (which exactly remp fun does). I can switch from t-test to Welch's by simply setting the Welch parameter to TRUE, but the question is the same, is this bootstrapping approach correct? | |
| May 16, 2022 at 10:05 | comment | added | Tomás Navarro | @awhug I'll try to clarify the goal. Because the t-test requires that the sample come from normal distributions, the percentage of p-values < 0.05 does not have to be exactly 0.05 in the example. When performing a t-test on highly skewed distributions robustness is lost, meaning that the empirical type I error is not equal to the nominal alpha (0.05). Various threads discuss the issue. | |
| May 16, 2022 at 0:29 | comment | added | awhug |
It's hard to tell what you're trying to achieve with this analysis. You're forcing the mean of x and y to be 0, and then resampling each with replacement (assuming this is what remp does) and testing the difference between them (which by definition is now 0) using a Student's t-test. Having ~5% of p-values <= .05 is exactly what we'd expect here. If you want a test of the difference that's robust to data from lightly skewed distributions, why not use the regular Welch test and skip the bootstrap? Unless you want something different?
|
|
| S May 13, 2022 at 20:26 | review | First questions | |||
| May 14, 2022 at 2:23 | |||||
| S May 13, 2022 at 20:26 | history | asked | Tomás Navarro | CC BY-SA 4.0 |