Skip to main content
Commonmark migration
Source Link

The example you mention is a little overly simplified for my taste. But the sentiment is right: correcting for collinearity is not a post-hoc procedure. This is guaranteed to lead to data dredging, since the result of collinearity is variance inflation, and adding or removing factors to get significant results (i.e. from reduced variance) is not a statistically sound procedure.

To dive into the example more, I think we have to distinguish randomized experimental studies and pseudo-randomized or observational studies.

randomized approaches

#randomized approaches BasicallyBasically, the examples suggests in both scenarios: use experimental or pseudo-experimental design to reduce collinearity of factors. Randomization should eliminate collinearity. This is called the randomization assumption. Blocking or blocked randomization, or blocked design, ensures that covariates are balanced exactly rather than probabilistically (i.e. in their expected value). Similar to this is re-randomization; you can re-roll your die to obtain covariate levels which are balanced if the cohort design is not staggered. Other blocking strategies have been discussed elsewhere, like Latin Squares, and so on.

pseudo-randomized approaches

In observational studies, then, a new field of causal modeling has emphasized the rationale for adjusting for certain variables in the analysis. With regards to whether it causes collinearity or not, we say, "So what?" And include it anyway. For instance, social depravity is highly correlated with the likelihood of child smoking and it is also highly correlated with children's health. When you adjust for its influence in measures of association between smoking and health, you find a somewhat attenuated estimate. This is because, by not accounting for social depravity, smoking is to some extent a measure of the hardship of life: low income, stable housing, adequate food, and so on, as well as the direct effect of smoking on health.

Often studies aren't randomized, so that several confounding variables may affect the association between a pseudo-treatment and an outcome. Here, a pseudo-treatment is something that can't be randomized for one reason or another, like smoking in children. Matching or paired design based on exposed and unexposed individuals in strata defined by confounders would result in the confounders being orthogonal to the exposure. The distribution of confounders then is balanced between exposed and unexposed participants (you may also categorize the exposure into quantiles or such and obtained matched samples there).

Related to this, you might consider using propensity matching. The concept was born out of an approach to blend observational and experimental design. It turns out, if you can construct a prediction model for the likelihood of receipt of exposure, based on measured confounding variables, that model produces a propensity score. By matching equally likely cases and controls, you may obtain a subsample of the observational sample which is "balanced" in some sense. But again, choice of confounders post hoc is often a bad idea, because we rarely just "conveniently" measure them, they must be pre-specified. Omitting confounders results in poor performance and bias of propensity matched analyses.

However, both of these approaches are equivalent to just adjusting for the confounders in the model.

So in both cases, collinearity is something which is addressed a priori to data collection. The question, then, shouldn't be how to prevent collinearity from a data analysis perspective. It's simply to make a note of it, and account for the reduction in power. If a factor is the correct thing to adjust for in a model, you must do it if you can. If it renders results non-significant when an association would be expected, you may hypothesize that variance inflation overspent your power, and additionally report unadjusted associations, commenting on their differences. This would be an inconclusive finding, however.

The example you mention is a little overly simplified for my taste. But the sentiment is right: correcting for collinearity is not a post-hoc procedure. This is guaranteed to lead to data dredging, since the result of collinearity is variance inflation, and adding or removing factors to get significant results (i.e. from reduced variance) is not a statistically sound procedure.

To dive into the example more, I think we have to distinguish randomized experimental studies and pseudo-randomized or observational studies.

#randomized approaches Basically, the examples suggests in both scenarios: use experimental or pseudo-experimental design to reduce collinearity of factors. Randomization should eliminate collinearity. This is called the randomization assumption. Blocking or blocked randomization, or blocked design, ensures that covariates are balanced exactly rather than probabilistically (i.e. in their expected value). Similar to this is re-randomization; you can re-roll your die to obtain covariate levels which are balanced if the cohort design is not staggered. Other blocking strategies have been discussed elsewhere, like Latin Squares, and so on.

pseudo-randomized approaches

In observational studies, then, a new field of causal modeling has emphasized the rationale for adjusting for certain variables in the analysis. With regards to whether it causes collinearity or not, we say, "So what?" And include it anyway. For instance, social depravity is highly correlated with the likelihood of child smoking and it is also highly correlated with children's health. When you adjust for its influence in measures of association between smoking and health, you find a somewhat attenuated estimate. This is because, by not accounting for social depravity, smoking is to some extent a measure of the hardship of life: low income, stable housing, adequate food, and so on, as well as the direct effect of smoking on health.

Often studies aren't randomized, so that several confounding variables may affect the association between a pseudo-treatment and an outcome. Here, a pseudo-treatment is something that can't be randomized for one reason or another, like smoking in children. Matching or paired design based on exposed and unexposed individuals in strata defined by confounders would result in the confounders being orthogonal to the exposure. The distribution of confounders then is balanced between exposed and unexposed participants (you may also categorize the exposure into quantiles or such and obtained matched samples there).

Related to this, you might consider using propensity matching. The concept was born out of an approach to blend observational and experimental design. It turns out, if you can construct a prediction model for the likelihood of receipt of exposure, based on measured confounding variables, that model produces a propensity score. By matching equally likely cases and controls, you may obtain a subsample of the observational sample which is "balanced" in some sense. But again, choice of confounders post hoc is often a bad idea, because we rarely just "conveniently" measure them, they must be pre-specified. Omitting confounders results in poor performance and bias of propensity matched analyses.

However, both of these approaches are equivalent to just adjusting for the confounders in the model.

So in both cases, collinearity is something which is addressed a priori to data collection. The question, then, shouldn't be how to prevent collinearity from a data analysis perspective. It's simply to make a note of it, and account for the reduction in power. If a factor is the correct thing to adjust for in a model, you must do it if you can. If it renders results non-significant when an association would be expected, you may hypothesize that variance inflation overspent your power, and additionally report unadjusted associations, commenting on their differences. This would be an inconclusive finding, however.

The example you mention is a little overly simplified for my taste. But the sentiment is right: correcting for collinearity is not a post-hoc procedure. This is guaranteed to lead to data dredging, since the result of collinearity is variance inflation, and adding or removing factors to get significant results (i.e. from reduced variance) is not a statistically sound procedure.

To dive into the example more, I think we have to distinguish randomized experimental studies and pseudo-randomized or observational studies.

randomized approaches

Basically, the examples suggests in both scenarios: use experimental or pseudo-experimental design to reduce collinearity of factors. Randomization should eliminate collinearity. This is called the randomization assumption. Blocking or blocked randomization, or blocked design, ensures that covariates are balanced exactly rather than probabilistically (i.e. in their expected value). Similar to this is re-randomization; you can re-roll your die to obtain covariate levels which are balanced if the cohort design is not staggered. Other blocking strategies have been discussed elsewhere, like Latin Squares, and so on.

pseudo-randomized approaches

In observational studies, then, a new field of causal modeling has emphasized the rationale for adjusting for certain variables in the analysis. With regards to whether it causes collinearity or not, we say, "So what?" And include it anyway. For instance, social depravity is highly correlated with the likelihood of child smoking and it is also highly correlated with children's health. When you adjust for its influence in measures of association between smoking and health, you find a somewhat attenuated estimate. This is because, by not accounting for social depravity, smoking is to some extent a measure of the hardship of life: low income, stable housing, adequate food, and so on, as well as the direct effect of smoking on health.

Often studies aren't randomized, so that several confounding variables may affect the association between a pseudo-treatment and an outcome. Here, a pseudo-treatment is something that can't be randomized for one reason or another, like smoking in children. Matching or paired design based on exposed and unexposed individuals in strata defined by confounders would result in the confounders being orthogonal to the exposure. The distribution of confounders then is balanced between exposed and unexposed participants (you may also categorize the exposure into quantiles or such and obtained matched samples there).

Related to this, you might consider using propensity matching. The concept was born out of an approach to blend observational and experimental design. It turns out, if you can construct a prediction model for the likelihood of receipt of exposure, based on measured confounding variables, that model produces a propensity score. By matching equally likely cases and controls, you may obtain a subsample of the observational sample which is "balanced" in some sense. But again, choice of confounders post hoc is often a bad idea, because we rarely just "conveniently" measure them, they must be pre-specified. Omitting confounders results in poor performance and bias of propensity matched analyses.

However, both of these approaches are equivalent to just adjusting for the confounders in the model.

So in both cases, collinearity is something which is addressed a priori to data collection. The question, then, shouldn't be how to prevent collinearity from a data analysis perspective. It's simply to make a note of it, and account for the reduction in power. If a factor is the correct thing to adjust for in a model, you must do it if you can. If it renders results non-significant when an association would be expected, you may hypothesize that variance inflation overspent your power, and additionally report unadjusted associations, commenting on their differences. This would be an inconclusive finding, however.

mentioned propensity scores, bolded approaches, organized response
Source Link
AdamO
  • 64.8k
  • 6
  • 135
  • 273

Basically#randomized approaches Basically, the examples suggests in both scenarios: use experimental or pseudo-experimental design to reduce collinearity of factors. In randomized studies, you obtain covariate balance throughRandomization should eliminate collinearity. This is called the randomization assumptionrandomization assumption. BlockedBlocking or blocked randomization, or blocked design, ensures that covariates are balanced exactly rather than probabilistically (i.e. in their expected value). Similar to this is re-randomization;re-randomization; you can re-roll your die to obtain covariate levels which are balanced if the cohort design is not staggered. Other blocking strategies have been discussed elsewhere, like Latin Squares, and so on.

Often studies aren't randomized, so that several confounding variables may affect the association between a pseudo-treatment and an outcome. Here, a pseudo-treatment is something that can't be randomized for one reason or another, like smoking in children. Interestingly, pseudo-experimental design still only affects efficiency in this case: stratified sampling by risk factors--such as age, or social depravity based on neighborhood SES indicators in our smoking example--ensures that you sample a "broader" portrait of the general population without having to sample many people. This will not prevent collinearity as the example suggests, despite following a pseudo-experimental design.

pseudo-randomized approaches

In observational studies, then, a new field of causal modeling has emphasized the rationale for adjusting for certain variables in the analysis. With regards to whether it causes collinearity or not, we say, "So what?" And include it anyway. For instance, social depravity is highly correlated with the likelihood of child smoking and it is also highly correlated with children's health. When you adjust for its influence in measures of association between smoking and health, you find a somewhat attenuated estimate. This is because, by not accounting for social depravity, smoking is to some extent a measure of the hardship of life: low income, stable housing, adequate food, and so on, as well as the direct effect of smoking on health.

Often studies aren't randomized, so that several confounding variables may affect the association between a pseudo-treatment and an outcome. Here, a pseudo-treatment is something that can't be randomized for one reason or another, like smoking in children. Matching or paired design based on exposed and unexposed individuals in strata defined by confounders would result in the confounders being orthogonal to the exposure. The distribution of confounders then is balanced between exposed and unexposed participants (you may also categorize the exposure into quantiles or such and obtained matched samples there).

Related to this, you might consider using propensity matching. The concept was born out of an approach to blend observational and experimental design. It turns out, if you can construct a prediction model for the likelihood of receipt of exposure, based on measured confounding variables, that model produces a propensity score. By matching equally likely cases and controls, you may obtain a subsample of the observational sample which is "balanced" in some sense. But again, choice of confounders post hoc is often a bad idea, because we rarely just "conveniently" measure them, they must be pre-specified. Omitting confounders results in poor performance and bias of propensity matched analyses.

However, both of these approaches are equivalent to just adjusting for the confounders in the model.

Basically, the examples suggests in both scenarios: use experimental or pseudo-experimental design to reduce collinearity of factors. In randomized studies, you obtain covariate balance through the randomization assumption. Blocked randomization, or blocked design, ensures that covariates are balanced exactly rather than probabilistically (i.e. in their expected value). Similar to this is re-randomization; you can re-roll your die to obtain covariate levels which are balanced if the cohort design is not staggered. Other blocking strategies have been discussed elsewhere, like Latin Squares, and so on.

Often studies aren't randomized, so that several confounding variables may affect the association between a pseudo-treatment and an outcome. Here, a pseudo-treatment is something that can't be randomized for one reason or another, like smoking in children. Interestingly, pseudo-experimental design still only affects efficiency in this case: stratified sampling by risk factors--such as age, or social depravity based on neighborhood SES indicators in our smoking example--ensures that you sample a "broader" portrait of the general population without having to sample many people. This will not prevent collinearity as the example suggests, despite following a pseudo-experimental design.

In observational studies, then, a new field of causal modeling has emphasized the rationale for adjusting for certain variables in the analysis. With regards to whether it causes collinearity or not, we say, "So what?" And include it anyway. For instance, social depravity is highly correlated with the likelihood of child smoking and it is also highly correlated with children's health. When you adjust for its influence in measures of association between smoking and health, you find a somewhat attenuated estimate. This is because, by not accounting for social depravity, smoking is to some extent a measure of the hardship of life: low income, stable housing, adequate food, and so on, as well as the direct effect of smoking on health.

#randomized approaches Basically, the examples suggests in both scenarios: use experimental or pseudo-experimental design to reduce collinearity of factors. Randomization should eliminate collinearity. This is called the randomization assumption. Blocking or blocked randomization, or blocked design, ensures that covariates are balanced exactly rather than probabilistically (i.e. in their expected value). Similar to this is re-randomization; you can re-roll your die to obtain covariate levels which are balanced if the cohort design is not staggered. Other blocking strategies have been discussed elsewhere, like Latin Squares, and so on.

pseudo-randomized approaches

In observational studies, then, a new field of causal modeling has emphasized the rationale for adjusting for certain variables in the analysis. With regards to whether it causes collinearity or not, we say, "So what?" And include it anyway. For instance, social depravity is highly correlated with the likelihood of child smoking and it is also highly correlated with children's health. When you adjust for its influence in measures of association between smoking and health, you find a somewhat attenuated estimate. This is because, by not accounting for social depravity, smoking is to some extent a measure of the hardship of life: low income, stable housing, adequate food, and so on, as well as the direct effect of smoking on health.

Often studies aren't randomized, so that several confounding variables may affect the association between a pseudo-treatment and an outcome. Here, a pseudo-treatment is something that can't be randomized for one reason or another, like smoking in children. Matching or paired design based on exposed and unexposed individuals in strata defined by confounders would result in the confounders being orthogonal to the exposure. The distribution of confounders then is balanced between exposed and unexposed participants (you may also categorize the exposure into quantiles or such and obtained matched samples there).

Related to this, you might consider using propensity matching. The concept was born out of an approach to blend observational and experimental design. It turns out, if you can construct a prediction model for the likelihood of receipt of exposure, based on measured confounding variables, that model produces a propensity score. By matching equally likely cases and controls, you may obtain a subsample of the observational sample which is "balanced" in some sense. But again, choice of confounders post hoc is often a bad idea, because we rarely just "conveniently" measure them, they must be pre-specified. Omitting confounders results in poor performance and bias of propensity matched analyses.

However, both of these approaches are equivalent to just adjusting for the confounders in the model.

Source Link
AdamO
  • 64.8k
  • 6
  • 135
  • 273

The example you mention is a little overly simplified for my taste. But the sentiment is right: correcting for collinearity is not a post-hoc procedure. This is guaranteed to lead to data dredging, since the result of collinearity is variance inflation, and adding or removing factors to get significant results (i.e. from reduced variance) is not a statistically sound procedure.

To dive into the example more, I think we have to distinguish randomized experimental studies and pseudo-randomized or observational studies.

Basically, the examples suggests in both scenarios: use experimental or pseudo-experimental design to reduce collinearity of factors. In randomized studies, you obtain covariate balance through the randomization assumption. Blocked randomization, or blocked design, ensures that covariates are balanced exactly rather than probabilistically (i.e. in their expected value). Similar to this is re-randomization; you can re-roll your die to obtain covariate levels which are balanced if the cohort design is not staggered. Other blocking strategies have been discussed elsewhere, like Latin Squares, and so on.

Often studies aren't randomized, so that several confounding variables may affect the association between a pseudo-treatment and an outcome. Here, a pseudo-treatment is something that can't be randomized for one reason or another, like smoking in children. Interestingly, pseudo-experimental design still only affects efficiency in this case: stratified sampling by risk factors--such as age, or social depravity based on neighborhood SES indicators in our smoking example--ensures that you sample a "broader" portrait of the general population without having to sample many people. This will not prevent collinearity as the example suggests, despite following a pseudo-experimental design.

In observational studies, then, a new field of causal modeling has emphasized the rationale for adjusting for certain variables in the analysis. With regards to whether it causes collinearity or not, we say, "So what?" And include it anyway. For instance, social depravity is highly correlated with the likelihood of child smoking and it is also highly correlated with children's health. When you adjust for its influence in measures of association between smoking and health, you find a somewhat attenuated estimate. This is because, by not accounting for social depravity, smoking is to some extent a measure of the hardship of life: low income, stable housing, adequate food, and so on, as well as the direct effect of smoking on health.

So in both cases, collinearity is something which is addressed a priori to data collection. The question, then, shouldn't be how to prevent collinearity from a data analysis perspective. It's simply to make a note of it, and account for the reduction in power. If a factor is the correct thing to adjust for in a model, you must do it if you can. If it renders results non-significant when an association would be expected, you may hypothesize that variance inflation overspent your power, and additionally report unadjusted associations, commenting on their differences. This would be an inconclusive finding, however.