I am working to build a model using logistic regression. There is one variable which has strong predictive power. But based on business rule, I cannot include this variable in the model. On the other hand, there are some variables which are insignificant in the model. But from business decision, they are added in the model.

So my question is how the model will be impacted if I remove an important variable? And how will it be impacted if I add some insignificant variables? Is there a way to measure the impact?

  • 1
    $\begingroup$ What is this "business rule"? Any rule which stops you from including important variables - well, it might be a rule worth breaking. $\endgroup$
    – Peter Flom
    Nov 27 '12 at 21:19
  • $\begingroup$ It's kind of complicated to explain. This is a project for P&C insurance industry where there are some limitations for variables. Including some sensitive variables in the model may cause legal issue. So it’s hard to break this rule. $\endgroup$ Nov 27 '12 at 21:40
  • 2
    $\begingroup$ Variables like race and gender are often excluded explicitly for legal considerations, but then are frequently "snuck in" by including highly correlated variables as proxies. $\endgroup$
    – dimitriy
    Nov 27 '12 at 21:43
  • 3
    $\begingroup$ Why are you building this model? Do you want to be able to predict something in the future, or are you trying to understand why something happens, or do you simply want to describe the strength of the relationship b/t some variables, or something else? What you should do depends critically on what your goals are. $\endgroup$ Nov 27 '12 at 21:57

At the risk of giving almost worthless advice, the only right answer is: It depends!

You should talk to your client/boss and ask why the business rules mandate including/excluding certain variables. In some cases, there might be a strict legal requirement. For example, in the US, your recruiting system definitely should not consider applicant's race, gender or marital status when deciding which applicants to hire. This could be---and probably are---strong predictors for some industries, but to include them would open your company up to a world of legal hurt. Similarly, other regulations may require that certain factors be considered in certain decisions, even if they're mostly useless.

Other times, the business rules might reflect something more negotiable. You clearly can't use tomorrow's numbers to predict today's sales (causality and all). However, depending on how your company collects and distributes its data, you might not be able to use yesterday's sales to predict today's either. Maybe the sales numbers are only updated twice a week! Similarly, some variables might be costly to measure regularly and your company is trying to control expenses. Depending on the strength of the predictor and the importance of your model, you might be able to lobby for different/additional data collection that would allow you to use these predictors.

Finally, some business rules are just dumb: they're based on bad assumptions, arise from antiquated procedures, or whatever. Perhaps they could be changed if you ask.

Obviously, removing a powerful predictor from your model will make it perform worse; including irrelevant predictors might be more tolerable: hopefully all of their coefficients will be near zero. Alternately, you could slap on some feature selection step which "automatically" prunes them.

I'd try to discern the reasons behind the business rules and offer your boss/client several options.


I would like to go with the advices @MattKrause with more highlight on the feature selection part. Once you check the business rules and generate all the representative variables, you are advised to partition data into a training, validation and testing sets.

There are many approaches for feature selection. However, if you are more interested with the predctivity of the model, you may go with the Wrapper feature selection model. You wrap around your regression model any search mechanism to select different feature subsets and evaluate the performance e.g. using MSE over the validation part of data.

Finally, keep in mind that relevance does not imply optimal and vice versa. Thus, a feature that looks relevant is not necessarily one that help predictivity of the model. This rule differs based on the characteristics of the chosen regression model. In addition, even if a feature is of low relevance (i.e. low correlation with the target), it might complement another feature and improve predictivity !!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.