0
$\begingroup$

Let's just say that in my field several studies find some significant associations between a variety of CEO attributes and some organizational policy (binary categorical dependent variable). Many use logistic regression and then infer that any statistically significant attribute is a causal factor for why CEOs are more likely to adopt certain policies.

What I have found however is that there may be reason to question whether the policies are even a product of CEO decision-making. A through review of the cases reveals that in ~30% of them, the policies are actually initiated by middle-management and not the CEOs.

So my hypothesis is that by including all cases, as previous research does, results in biased estimates. I predict that when re-running their analyses with only relevant positive cases where CEO decision-making mattered, we should then see different effects or even non-significant associations with the various CEO attributes. I am struggling, however, with three things:

  1. So how can I better explain the problem in more formal methodological terms? I know measurement error on the DV does not bias estimates for the IVs in interval data, but this doesn't really make sense in my case since the DV is categorical. Is this misclassification? measurement error, Type 1 error, or something different?

  2. What is the best way to test this hypothesis using statistical methods?

In brief, my strategy is to run two (logistic) models, one where all positive cases are included and another where only relevant positive cases are included, and then compare the models based on change in effect sizes, significance levels, and McFaddens's pseudo R-squared. Only problem is I know the significance level will decrease regardless because there will be less positive observations.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Measurement error of course introduces bias. If you systematically measure the phenomenon of interest wrong, you systematically get the wrong results.

To test your hypothesis, create a dummy variable that you could make more refined later, called middle_management or something. It takes on the value of 1 when you know middle management initiated the policy and 0 when you know the CEO took it on. This acts as a "control" for this "confounder".

Your strategy is not great. You cant just model the positive cases, you have to model the positive and negative cases together. For one, the model wont run with just a bunch of 1s.

To offer more help; describe your data set, its variables and values, and how big it is. The research question would be helpful, too.

Improve this answer
$\endgroup$
8
  • $\begingroup$ thanks for your help. Yes I wasn't too clear on the strategy, but I essentially thought I could run two models on the same data (that includes DV = 0 observations), but the second model would only include positive observations where CEO = 1 and exclude positive observations where middle_management = 1. Although, I wonder if your method would be better as it seems simpler and more intuitive? Research question is "to what extent can we make valid causal inferences about CEO attributes using policy x data?" $\endgroup$
    – BobC
    Jan 25, 2021 at 13:45
  • $\begingroup$ Dataset has ~300 positive cases and ~10000 negative cases (WIP). Of the positive cases, I only expect that ~150 are CEO = 1. There are about 4 main IV's (CEO attributes) and about 5 controls (organizational factors) $\endgroup$
    – BobC
    Jan 25, 2021 at 13:51
  • $\begingroup$ Think of a single model as testing a single hypothesis in a strict sense. Changing the model changes your hypothesis, presuming the model was specified carefully to match your hypothesis. In a less strict sense, you can compare nested models by swapping/adding variables and looking at its influence, comparing the model to itself. But using one set of data to compare it to another set of data is very different. I would probably follow my intuition where you keep the DV binary, use logistic regression, and add a dummy variable. $\endgroup$
    – John Stud
    Jan 25, 2021 at 13:55
  • $\begingroup$ What statistical software are you using? Since you have a very large class imbalance, you will likely want to weight the classes appropriately. sklearn in python makes this very easy for you: scikit-learn.org/stable/modules/generated/… I am unsure how to do the same in R with GLM. $\endgroup$
    – John Stud
    Jan 25, 2021 at 13:58
  • $\begingroup$ Thank you so much for your help. I use R. So just so I understand, in your way I am still modeling CEO attribute x when Policy = 1, but just controlling for CEO decision-making relevance. So I interpret that the effect of all the IV's as only when middle_management = 0? So if my hypothesis is correct (no relationship between IVs when controlling for CEO decision-making relevance), than this suggests existing findings are biased due to misclassification error? $\endgroup$
    – BobC
    Jan 25, 2021 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.