19
$\begingroup$

I'm working on logistic regression model. I checked the summary of the model which is built on 5 independent variables out which one is not significant with a P-value of 0.74.I wish to know that do we directly remove the variable or is there any other way to check for it's significance?

A senior of mine suggested to do logarithmic transformation of the insignificant variable & look for correlation then. Will that count towards checking it's significance.

model <- glm(Buy ~ a_score + b_score+ c_score+lb+p, data = history, family = binomial)

All variables come out to be significant with 2 or 3 stars apart from a_score which is shown insignificant.

Improve this question
$\endgroup$
1
  • $\begingroup$ What is the goal of your model building exercise? Are you interested in inferential statistics, e.g., for an academic article, or in prediction? $\endgroup$
    – Stephan Kolassa
    Jun 20, 2019 at 11:53

3 Answers 3

Reset to default
63
$\begingroup$

Let me first ask this: What is the goal of the model? If you are only interested in predicting if a customer will buy, then statistcal hypothesis tests really aren't your main concern. Instead, you should be externally validating your model via a validation/test prodecedure on unseen data.

If, instead, you are interested in examining which factors contribute to the probability of a customer buying, then there is no need to remove variables which fail to reject the null (especially in a stepwise sort of manner). Presumably, you included a variable in your model because you thought (from past experience or expert opinion) that it played an important part in a customer deciding if they will buy. That the variable failed to reject the null doesn't make your model a bad one, it just means that your sample didin't detect an effect of that variable. That's perfectly ok.

Improve this answer
$\endgroup$
5
  • 4
    $\begingroup$ Upvoted for excellence of the answer. $\endgroup$
    – James Phillips
    Jun 18, 2019 at 16:42
  • 9
    $\begingroup$ +1 Removing predictors potentially related to outcome (even if "insignificant") is tricky in logistic regression, given its inherent omitted-variable bias. Removing a predictor related to outcome can lead to bias in the estimates of the coefficients of the retained predictors, even if the retained predictors aren't correlated with the removed predictor. $\endgroup$
    – EdM
    Jun 18, 2019 at 17:01
  • 4
    $\begingroup$ This is indeed a very clear answer. $\endgroup$
    – gented
    Jun 19, 2019 at 11:41
  • $\begingroup$ To add on to this, for those variables deemed insignificant, there wasn't any evidence to suggest the coefficient was that much different from 0. So in leaving those variables in the model, the weight of that coefficient is very small and should have a negligible impact on your prediction. $\endgroup$
    – blacksite
    Oct 16, 2020 at 13:14
  • 2
    $\begingroup$ @blacksite That isn't necessarily true. You can imagine a binary variable, $z$, with a large coefficient which fails to reject the null. In this case $y\vert z=0$ is could potentially be very different than $y \vert z=1$. Here is an example set.seed(3) N = 10000 x = rnorm(N) z = rbinom(N,1, 0.001) y = 2*x + 1 + 10*z + rnorm(N, 0, 10) model = lm(y ~ x + z) summary(model) $\endgroup$
    – Demetri Pananos
    Oct 16, 2020 at 14:39
2
$\begingroup$

Have a look at the help pages for step(), drop1() and add1(). These will help you to add/remove variables based on AIC. However, all such methods are somewhat flawed in their path dependence. A better way would be to use the functions in the penalised or glmnet package to perform a lasso regression.

Improve this answer
$\endgroup$
1
$\begingroup$

What is the correlations among the independent variables? This is less important for pure prediction, but if you want to gain some inferential information it is important that the independent variables be fairly uncorrelated. Typically, when you use logistic regression in a business setting, both inferential information about the variables used along with a good prediction are what stakeholders are looking for.

Additionally, another good reason to remove variables is for model parsimony. Some reasons for this is for internal review purposes, legal regulation, and ease of implementation. These lead to it being highly desirable to find the smallest set of variables that give good business information and good predictions. For example, if you are developing a credit model, every variable is subject to legal review, every variable has to be available and immediately return values when called to score the loan, and the stakeholders (who usually are not versed in model building) tend to not want to look at complicated models loaded with variables.

It may be also helpful to try a random forest to get some idea of variable importance and also to check the predictive power with and without all the variables.

Finally, you should have a good reason for transforming a variable. Throwing every transformation against a variable until you find one that gives you the result you want is a good way to get an overfit model that performs poorly on new data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.