Skip to main content
replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
Source Link
  1. Those standard deviations aren't so badly different.

  2. Since $n_1=41000$, and the standard deviations aren't very large, even if the variances were very different, it wouldn't matter.

You could even treat the mean of the first sample as fixed (it almost is) and do a one sample t-test.

  1. The skewness likely won't matter much either, unless it's quite strong in the smaller sample. (you say 'skews above 10' ... but that doesn't really say how big they are. If, say the skewness in the smaller sample is less than 20, the distribution of the mean should still be close to normal, and between CLT for the numerator and using Slutsky's theorem for the rest of the statistic, it should be close to normal)

--

The Welch test should be okay.

Another alternative is to consider a permutation test (the standard deviations aren't all that different) or a bootstrap test. They'll likely give very similar results to what you already have.


Edit: (Answering follow-up question from comments)

Well, sure. The way to tell if the difference is not so bad is to see how much impact ignoring it would have.

The relevant measures of impact are the significance level when $H_0$ is true and power when it's false, and more generally the shape of the power function (which can reveal issues like test bias). You can most easily calculate and compare power functions under various assumptions via simulation.

For example, I used simulation in parts of my answer to this related questionthis related question. I carried out those simulations in R.

So you can assume some population ratios of variances close to the one observed and see how badly it affects significance and power if you treat them as equal, and how close to the nominal significance you get if you use say the Welch approximation instead, as well as any impact on power.

  1. Those standard deviations aren't so badly different.

  2. Since $n_1=41000$, and the standard deviations aren't very large, even if the variances were very different, it wouldn't matter.

You could even treat the mean of the first sample as fixed (it almost is) and do a one sample t-test.

  1. The skewness likely won't matter much either, unless it's quite strong in the smaller sample. (you say 'skews above 10' ... but that doesn't really say how big they are. If, say the skewness in the smaller sample is less than 20, the distribution of the mean should still be close to normal, and between CLT for the numerator and using Slutsky's theorem for the rest of the statistic, it should be close to normal)

--

The Welch test should be okay.

Another alternative is to consider a permutation test (the standard deviations aren't all that different) or a bootstrap test. They'll likely give very similar results to what you already have.


Edit: (Answering follow-up question from comments)

Well, sure. The way to tell if the difference is not so bad is to see how much impact ignoring it would have.

The relevant measures of impact are the significance level when $H_0$ is true and power when it's false, and more generally the shape of the power function (which can reveal issues like test bias). You can most easily calculate and compare power functions under various assumptions via simulation.

For example, I used simulation in parts of my answer to this related question. I carried out those simulations in R.

So you can assume some population ratios of variances close to the one observed and see how badly it affects significance and power if you treat them as equal, and how close to the nominal significance you get if you use say the Welch approximation instead, as well as any impact on power.

  1. Those standard deviations aren't so badly different.

  2. Since $n_1=41000$, and the standard deviations aren't very large, even if the variances were very different, it wouldn't matter.

You could even treat the mean of the first sample as fixed (it almost is) and do a one sample t-test.

  1. The skewness likely won't matter much either, unless it's quite strong in the smaller sample. (you say 'skews above 10' ... but that doesn't really say how big they are. If, say the skewness in the smaller sample is less than 20, the distribution of the mean should still be close to normal, and between CLT for the numerator and using Slutsky's theorem for the rest of the statistic, it should be close to normal)

--

The Welch test should be okay.

Another alternative is to consider a permutation test (the standard deviations aren't all that different) or a bootstrap test. They'll likely give very similar results to what you already have.


Edit: (Answering follow-up question from comments)

Well, sure. The way to tell if the difference is not so bad is to see how much impact ignoring it would have.

The relevant measures of impact are the significance level when $H_0$ is true and power when it's false, and more generally the shape of the power function (which can reveal issues like test bias). You can most easily calculate and compare power functions under various assumptions via simulation.

For example, I used simulation in parts of my answer to this related question. I carried out those simulations in R.

So you can assume some population ratios of variances close to the one observed and see how badly it affects significance and power if you treat them as equal, and how close to the nominal significance you get if you use say the Welch approximation instead, as well as any impact on power.

added 746 characters in body
Source Link
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k
  1. Those standard deviations aren't so badly different.

  2. Since $n_1=41000$, and the standard deviations aren't very large, even if the variances were very different, it wouldn't matter.

You could even treat the mean of the first sample as fixed (it almost is) and do a one sample t-test.

  1. The skewness likely won't matter much either, unless it's quite strong in the smaller sample. (you say 'skews above 1010' ... but that doesn't really say how big they are. If, say the skewness in the smallsmaller sample is less than 20, the distribution of the mean should still be close to normal, and between CLT for the numerator and using Slutsky's theorem for the rest of the statistic, it should be close to normal)

--

The Welch test should be okay.

Another alternative is to consider a permutation test (the standard deviations aren't all that different) or a bootstrap test. They'll likely give very similar results to what you already have.


Edit: (Answering follow-up question from comments)

Well, sure. The way to tell if the difference is not so bad is to see how much impact ignoring it would have.

The relevant measures of impact are the significance level when $H_0$ is true and power when it's false, and more generally the shape of the power function (which can reveal issues like test bias). You can most easily calculate and compare power functions under various assumptions via simulation.

For example, I used simulation in parts of my answer to this related question. I carried out those simulations in R.

So you can assume some population ratios of variances close to the one observed and see how badly it affects significance and power if you treat them as equal, and how close to the nominal significance you get if you use say the Welch approximation instead, as well as any impact on power.

  1. Those standard deviations aren't so badly different.

  2. Since $n_1=41000$, and the standard deviations aren't very large, even if the variances were very different, it wouldn't matter.

You could even treat the mean of the first sample as fixed (it almost is) and do a one sample t-test.

  1. The skewness likely won't matter much either, unless it's quite strong in the smaller sample. (you say 'skews above 10 ... but that doesn't really say how big they are. If, say the skewness in the small sample is less than 20, the distribution of the mean should still be close to normal)

--

The Welch test should be okay.

Another alternative is to consider a permutation test (the standard deviations aren't all that different) or a bootstrap test. They'll likely give very similar results to what you already have.

  1. Those standard deviations aren't so badly different.

  2. Since $n_1=41000$, and the standard deviations aren't very large, even if the variances were very different, it wouldn't matter.

You could even treat the mean of the first sample as fixed (it almost is) and do a one sample t-test.

  1. The skewness likely won't matter much either, unless it's quite strong in the smaller sample. (you say 'skews above 10' ... but that doesn't really say how big they are. If, say the skewness in the smaller sample is less than 20, the distribution of the mean should still be close to normal, and between CLT for the numerator and using Slutsky's theorem for the rest of the statistic, it should be close to normal)

--

The Welch test should be okay.

Another alternative is to consider a permutation test (the standard deviations aren't all that different) or a bootstrap test. They'll likely give very similar results to what you already have.


Edit: (Answering follow-up question from comments)

Well, sure. The way to tell if the difference is not so bad is to see how much impact ignoring it would have.

The relevant measures of impact are the significance level when $H_0$ is true and power when it's false, and more generally the shape of the power function (which can reveal issues like test bias). You can most easily calculate and compare power functions under various assumptions via simulation.

For example, I used simulation in parts of my answer to this related question. I carried out those simulations in R.

So you can assume some population ratios of variances close to the one observed and see how badly it affects significance and power if you treat them as equal, and how close to the nominal significance you get if you use say the Welch approximation instead, as well as any impact on power.

Source Link
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k

  1. Those standard deviations aren't so badly different.

  2. Since $n_1=41000$, and the standard deviations aren't very large, even if the variances were very different, it wouldn't matter.

You could even treat the mean of the first sample as fixed (it almost is) and do a one sample t-test.

  1. The skewness likely won't matter much either, unless it's quite strong in the smaller sample. (you say 'skews above 10 ... but that doesn't really say how big they are. If, say the skewness in the small sample is less than 20, the distribution of the mean should still be close to normal)

--

The Welch test should be okay.

Another alternative is to consider a permutation test (the standard deviations aren't all that different) or a bootstrap test. They'll likely give very similar results to what you already have.