Return to Answer

Gregg, do you mean for normal, heteroscedastic data? Your second paragraph seems to suggest so.

I added an answer to the original post you reference, where I suggested that if the data are normal but heteroscedastic, using generalized least squares provides the most flexible approach for dealing with the data features you mention. Not accounting explicitly for those features will lead to suboptimal and possibly misleading results, as you noticed in your own practice. How suboptimal or misleading the results may be will ultimately depend on the peculiarities of each data set.

A nice way to understand this would be to set up a simulation study where you can vary two factors: the number of groups and the extent to which the variability changes across groups. Then you could track the impact of these factors on the results of the test of differences between any of the means and the results of post-hoc comparisons between pairs of means when you use standard ANOVA (which ignores heteroscedasticity) versus gls (which accounts for heteroscedasticity).

Perhaps you could start your simulation exercise with a simple example with just 3 groups, where you keep the variability of the first two groups the same but change the variability of the third group by a factor f where f becomes increasingly large. This would allow you to see if and when that third group begins to dominate the results. (For simplicity, the differences in mean outcome values between each of the three groups could be kept the same, though you could look to see how the magnitude of the common difference plays with the magnitude of the variability in the third group.)

I think it would be hard to come up with a general assessment of what exactly might go wrong when heteroscedasticity is ignored, other than warning people that ignoring hetersoscedasticity is ill-advised when better methods for dealing with it exist.

Gregg, do you mean for normal, heteroscedastic data? Your second paragraph seems to suggest so.

I added an answer to the original post you reference, where I suggested that if the data are normal but heteroscedastic, using generalized least squares provides the most flexible approach for dealing with the data features you mention. Not accounting explicitly for those features will lead to suboptimal and possibly misleading results, as you noticed in your own practice. How suboptimal or misleading the results may be will ultimately depend on the peculiarities of each data set.

A nice way to understand this would be to set up a simulation study where you can vary two factors: the number of groups and the extent to which the variability changes across groups. Then you could track the impact of these factors on the results of the test of differences between any of the means and the results of post-hoc comparisons between pairs of means when you use standard ANOVA (which ignores heteroscedasticity) versus gls (which accounts for heteroscedasticity).

Perhaps you could start your simulation exercise with a simple example with just 3 groups, where you keep the variability of the first two groups the same but change the variability of the third group by a factor f where f becomes increasingly large. This would allow you to see if and when that third group begins to dominate the results. (For simplicity, the differences in mean outcome values between each of the three groups could be kept the same, though you could look to see how the magnitude of the common difference plays with the magnitude of the variability in the third group.)

I think it would be hard to come up with a general assessment of what exactly might go wrong when heteroscedasticity is ignored, other than warning people that ignoring heteroscedasticity is ill-advised when better methods for dealing with it exist.

Gregg, do you mean for normal, heteroscedastic data? Your second paragraph seems to suggest so.

I added an answer to the original post you reference, where I suggested that if the data are normal but heteroscedastic, using generalized least squares provides the most flexible approach for dealing with the data features you mention. Not accounting explicitely for those features will lead to suboptimal and possibly misleading results, as you noticed in your own practice. How suboptimal or misleading the results may be will ultimately depend on the peculiarities of each data set. A nice way to understand this would be to set up a simulation study where you can vary two factors: the number of groups and the extent to which the variability changes across groups. Then you could track the impact of these factors on the results of the test of differences between any of the means and the results of post-hoc comparisons between pairs of means when you use standard ANOVA (which ignores heteroscedasticity) versus gls (which accounts for heteroscedasticity). You could start with a simple example with just 3 groups, where you keep the variability of the first two groups the same but change the variability of the third group by a factor f where f becomes increasingly large. This would allow you to see if and when that third group begins to dominate the results. (For simplicity, the differences in mean outcome values between each of the three groups could be kept the same, though you could look to see how the magnitude of the common difference plays with the magnitude of the variability in the third group.)

I think it would be hard to come up with a general assessment of what exactly might go wrong when heteroscedasticity is ignored, other than warning people that ignoring hetersoscedasticity is ill-advised when better methods for dealing with it exist.

Gregg, do you mean for normal, heteroscedastic data? Your second paragraph seems to suggest so.

I added an answer to the original post you reference, where I suggested that if the data are normal but heteroscedastic, using generalized least squares provides the most flexible approach for dealing with the data features you mention. Not accounting explicitly for those features will lead to suboptimal and possibly misleading results, as you noticed in your own practice. How suboptimal or misleading the results may be will ultimately depend on the peculiarities of each data set.

A nice way to understand this would be to set up a simulation study where you can vary two factors: the number of groups and the extent to which the variability changes across groups. Then you could track the impact of these factors on the results of the test of differences between any of the means and the results of post-hoc comparisons between pairs of means when you use standard ANOVA (which ignores heteroscedasticity) versus gls (which accounts for heteroscedasticity).

Perhaps you could start your simulation exercise with a simple example with just 3 groups, where you keep the variability of the first two groups the same but change the variability of the third group by a factor f where f becomes increasingly large. This would allow you to see if and when that third group begins to dominate the results. (For simplicity, the differences in mean outcome values between each of the three groups could be kept the same, though you could look to see how the magnitude of the common difference plays with the magnitude of the variability in the third group.)

I think it would be hard to come up with a general assessment of what exactly might go wrong when heteroscedasticity is ignored, other than warning people that ignoring hetersoscedasticity is ill-advised when better methods for dealing with it exist.

Gregg, do you mean for normal, heteroscedastic data? Your second paragraph seems to suggest so.

I added an answer to the original post you reference, where I suggested that if the data are normal but heteroscedastic, using generalized least squares provides the most flexible approach for dealing with the data features you mention. Not accounting explicitely for those features will lead to suboptimal and possibly misleading results, as you noticed in your own practice. How suboptimal or misleading the results may be will ultimately depend on the peculiarities of each data set. A nice way to understand this would be to set up a simulation study where you can vary two factors: the number of groups and the extent to which the variability changes across groups. Then you could track the impact of these factors on the results of the test of differences between any of the means and the results of post-hoc comparisons between pairs of means when you use standard ANOVA (which ignores heteroscedasticity) versus gls (which accounts for heteroscedasticity).

Gregg, do you mean for normal, heteroscedastic data? Your second paragraph seems to suggest so.

I added an answer to the original post you reference, where I suggested that if the data are normal but heteroscedastic, using generalized least squares provides the most flexible approach for dealing with the data features you mention. Not accounting explicitely for those features will lead to suboptimal and possibly misleading results, as you noticed in your own practice. How suboptimal or misleading the results may be will ultimately depend on the peculiarities of each data set. A nice way to understand this would be to set up a simulation study where you can vary two factors: the number of groups and the extent to which the variability changes across groups. Then you could track the impact of these factors on the results of the test of differences between any of the means and the results of post-hoc comparisons between pairs of means when you use standard ANOVA (which ignores heteroscedasticity) versus gls (which accounts for heteroscedasticity). You could start with a simple example with just 3 groups, where you keep the variability of the first two groups the same but change the variability of the third group by a factor f where f becomes increasingly large. This would allow you to see if and when that third group begins to dominate the results. (For simplicity, the differences in mean outcome values between each of the three groups could be kept the same, though you could look to see how the magnitude of the common difference plays with the magnitude of the variability in the third group.)

I think it would be hard to come up with a general assessment of what exactly might go wrong when heteroscedasticity is ignored, other than warning people that ignoring hetersoscedasticity is ill-advised when better methods for dealing with it exist.