3
$\begingroup$

Firstly, I would like to ask you opinion about my model: I have a logistic regression model where the dependent variable is late payments (1=late/0=not late) and diesel prices and interest rates as the independent variables. I have checked for the VIF factor which less than 2 for each feature. These give p-values 0.000.

Secondly, would anyone know how could I improve the model? The pseudo $R^2$ value is just $0.02511.$ My goal is to find explanatory variables that could explain the late payments, and I tried several others, but they resulted with high VIF scores.

Improve this question
$\endgroup$
9
  • 3
    $\begingroup$ Use domain knowledge to improve your model. You do not say what late payments you are analyzing, and I find it intriguing to think of anything one could be late in paying where diesel prices would be a driving factor, but if you think that makes sense, so be it. Now you should just run through the same thought processes that suggested diesel prices as a useful predictor, and see what else they bring up. This may be useful: stats.stackexchange.com/q/222179/1352 $\endgroup$
    – Stephan Kolassa
    Commented Nov 16, 2022 at 21:39
  • 1
    $\begingroup$ Why do you involve VIF scores in variable selection? Is this some exercise for a course where you need to do something with VIF? What is the motivation behind this regression model and behind your approach? $\endgroup$
    – Sextus Empiricus
    Commented Nov 16, 2022 at 21:41
  • $\begingroup$ Thank you! The late payments are from transportation companies, and VIF scores are to double check that there is no multicollinearity issue. $\endgroup$
    – user373185
    Commented Nov 16, 2022 at 21:46
  • $\begingroup$ "The pseudo r^2 value is just 0.02511" Would you have expected that Diesel prices have a stronger statistical correlation with late payments? Could you explain why you believe that the low r^2 is a problem. From the statistician's point of view this is not clear why you expect a large r^2 value. $\endgroup$
    – Sextus Empiricus
    Commented Nov 16, 2022 at 21:46
  • 1
    $\begingroup$ 1) Which pseudo $R^2$ do you use? There are many. // 2) On what data do you measure your pseudo $R^2$ (in-sample, out-of-sample)? 3) I am concerned that this question is tantamount to asking how to do machine learning. Do you have a question that is more specific? 4) So what if there is multicollinearity? $\endgroup$
    – Dave
    Commented Nov 16, 2022 at 21:47

2 Answers 2

Reset to default
2
$\begingroup$

VIF should play no direct role. What are the two frequencies of the outcome variable? Why late vs not late as opposed to number of days late (with right censoring)? You might do some variable clustering (unsupervised learning) to group some collinear variables but don't exclude variables because of collinearity.

See my notes for some useful information about pseudo $R^2$ and adjusted versions of same: https://hbiostat.org/rmsc/lrm.html#quantifying-predictive-ability and the link contained there.

If you have done due diligence in using subject matter thinking to specify a model, and possibly done some unsupervised learning to live within the confines of your effective sample size, a low $R^2$ may be telling you that it's just hard to predict that outcome.

Improve this answer
$\endgroup$
0
$\begingroup$

In order to increase the performance of a given model we need to provide more information to our modelling procedure. That is done almost exclusive in two main ways: 1. we increase our sample size by adding new entries (either artificially by transformations or naturally by sampling additional points) so we know "more" about the underlying population and 2. by increasing the expressiveness/available features of our existing sample such that we know "more" about the existing sample. The former point is what we call "data augmentation" and the latter point is what we call "feature engineering". So if in this case the model is fixed to be a logistic regression model, we have to explore how these two options are applicable to our problem. While harsh, it has to be one of the two. Or we change the model entirely (e.g. a random forest) but that's another game. For example, what Stefan says, about the "use domain knowledge to improve your model" is effectively "expert feature engineering", while if you can get more samples is "real dataset augmentation".

As Frank mentioned, the VIF has little to do with the observed goodness-of-fit of a model. Multicollinearities are not a big problem in terms of predictive performance but more in terms of increased instability of predictor selection; see Performance of binary prediction models in high-correlation low-dimensional settings: a comparison of methods (2022) by Leeuwenberg et al. for more details. If within a regression framework, we are worried about multicollinearity and multiple variables being included, we can look into an elastic net logistic regression model at first instance. While not a panacea, the elastic net penalty will allow some degree of regularisation both for collinearities and multiple variables being included. Standard calculation of $R^2$ on the likelihood scale (McFadden, Cox and Snell, and Nagelkerke) work as before. As mentioned already though expectedly these will be even more unreliable than their unregularised counterparts in small samples; see Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small (2020) by Ripley et al. for more details.)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.