4
$\begingroup$

I'm actually trying to find the best explanatory variables in order to estimate the probability of deafult of the counterparties of my portfolio. After defined the Long List of variables, I'm testing each variable through a univariate logistic regression. Each variable is assessed considering three conditions:

  • statistically significant estimation, that is, p-value < 5%;
  • sign of the estimated coefficient (beta sign) coherent with economic expectations;
  • sufficient discriminatory power, that is, Somers’ D > 5%

However, focusing on the cost variable, firstly I tried to consider the variable by eliminating the observations where the amount of costs is equal to zero. The variable passes all the abovementioned criteria, but the Somer's D is lower (30%). Moreover, I don't think it's the best approach, since the zeroes are real zeroes and not missing values. For this reason, I considered to create a dummy variable in order to keep all the observations. I've created a dummy variable such that:

  • Dummy variable is equal to 1 when the positive continuous variable is equal to 0.
  • 0 otherwise.

So I've performed the following logit regression:

proc logistic data= want plots=ROC;
    model observed_default = continuous_variable dummy_variable  
run;

and the results are pretty good in terms of Somers' $D$ (60%) but the dummy variable is not significant (the $p$-value is equal to $0.35$).

How could I deal with this problem? Could I keep the dummy variable in the model even if it's not significant? I think the problem arises since the dummy variable is, by construction, collinear with the continuous variable.

Finally, after the univariate regressions, I'm going to estimate the multivariate one via the stepwise selection.

Improve this question
$\endgroup$
8
  • 5
    $\begingroup$ Could you tell us what "the problem" is? There's nothing inherently problematic about getting high p-values in a regression. It's also unclear how exactly you are employing your variables: what are the explanatory variables and what is the response variable? $\endgroup$
    – whuber ♦
    Commented Oct 13 at 18:47
  • 2
    $\begingroup$ That judgment is not always correct, because it depends on why you're doing this regression. Even when the judgment to remove a variable is correct, I would like to suggest the proper reaction should be "so what?," because what would be problematic about removing a variable? Finally, an argument can be made that you should be making the judgment using an omnibus test on both numerical expressions of your variable--the dummy and the continuous part--because conceptually you would either include both or neither. $\endgroup$
    – whuber ♦
    Commented Oct 13 at 20:37
  • 1
    $\begingroup$ You seem to be doing some sort of stepwise regression, and even more simplistic by considering univariate regression? Consider Stepwise AIC - Does there exist controversy surrounding this topic?. $\endgroup$
    – Sextus Empiricus
    Commented Oct 14 at 6:58
  • 2
    $\begingroup$ The question is a bit unclear. 1) You perform a weird analysis. 2) You say that there is a problem, but the problem is not made clear. 3) You ask for solutions, but it is not clear what you want to do with your data (What is there to optimize? We can not fill in the context for you. Questions like "find the best explanatory variables in order to estimate" are not standalone and require context to determine what sort of estimate is best). $\endgroup$
    – Sextus Empiricus
    Commented Oct 14 at 7:01
  • 1
    $\begingroup$ If the values of 0 are because there is no possibility of a non-0 value of a predictor, then (1) using a dummy for the possibility of a non-0 value makes sense and (2) it might be easier to interpret if you reversed the dummy so that a value of 1 represents the possibility of having a non-0 value. See this answer for details. In that situation you certainly need to include the dummy in your model so that you don't end up with omitted-variable bias. $\endgroup$
    – EdM
    Commented Oct 14 at 13:24

1 Answer 1

Reset to default
1
$\begingroup$

This question:

How could I deal with this problem?

assumes there is a problem. In my eyes, you have said nothing yet that immediately screams there is one, other than the fact that your $p$ value doesn't behave in a way you expect. Regarding this question:

Could I keep the dummy variable in the model even if it's not significant?

it is potentially problematic to add or remove variables post hoc purely because of a $p$ value. If my theory is that sunlight causes cancer, then run a test which shows the $p$ value is high, then it seems dishonest to then subsequently remove that variable, as we are not actually showing if our hypothesis was actually tested. It ends up snuggling up to the problematic practice of hypothesizing after results are known (HARKing) or $p$-hacking.

Your approach is not wrong in terms of dummy coding costs, but I think you need to be clear about what your model means by doing this. If we think there is some process behind this (the wording of your question implies this), then we need to make clear why this is the case in a published account of the analysis. You could use something like an omnibus test (as suggested in the comments) to compare your different treatments of the variable, but I think before that your justification needs to be clear.

With respect to the collinearity aspect, I do think that you should only use one or the other. They're both contributing very similar information, but one should take precedence over the other (in my opinion). Others may disagree, and I would be curious what their justification is to include both.

Edit

I noticed that you edited your question, which highlights a lot more context about your problem. For starters, never use stepwise regression. This has been discussed ad nauseam on this site, and is generally considered a poor technique for variable selection. If you want to employ more principled methods, you could consider penalized regression methods like ridge or lasso regression, but note that these methods are not a replacement for thinking about which subset of predictors matter, and may not be as useful for theoretical accounting of your variables. For prediction-only models, this will matter less.

It is also hazardous to simply fit several models with each different variables. The chances you get a nice result by pure chance increases dramatically with each iteration of model fitting.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Historically multivariate just means having many variables, And e.g. PCA remains a multivariate technique regardless of not being based on a distinction between dependent and independent variables. Other way round, regression is not now generally regarded as a multivariate method; or to be more precise you can talk about multivariate regression if (and only if) you have multiple dependent variables. As you say, a regression with one dependent variable and several independent variables can be called multiple or multivariable but either term is optional. $\endgroup$
    – Nick Cox
    Commented Oct 14 at 9:58
  • 1
    $\begingroup$ I would not call omission of insignificant predictors dishonest. I don't think it;s good practice, but there is a logic to it, which is that testing allows us to simplify a model. What would be dubious practice is concealing the decision-making in a report. That said, numerous projects try different models and report only the model that seems best and it can sometimes seem unduly pious to denounce such practices. Suppose an initial naive analysis makes it clear that transformation is needed. Is the researcher expected to explain the entire sequence of decisions? Reviewers might well object! $\endgroup$
    – Nick Cox
    Commented Oct 14 at 10:05
  • $\begingroup$ I think I agree with your points. Perhaps my beliefs about p value-based variable selection are driven by poor practices I see in my own field. I've amended my answer to soften my original wording, and just omit the stuff about multivariate analysis since it's not the primary point anyway. $\endgroup$
    – Shawn Hemelstrand
    Commented Oct 14 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.