2
$\begingroup$

I ran two logistic regression models, one with a dataset including outliers and one without outliers, with multiple predictors.

I checked each model's fit with the le Cessie – van Houwelingen – Copas – Hosmer unweighted sum of squares test for global goodness of fit from the rms package in R (following advice here).

model1 <- lrm(y ~ a + b + c + d, data1, method = "lrm.fit", model = TRUE, x = TRUE, y = TRUE, linear.predictors = TRUE, se.fit = FALSE)
residuals(model1, type = "gof")

For the model with outliers the p value was close to 0, indicating a lack of fit. For the model without outliers p value was 0.52, indicating that my model was not incorrect.

I then ran 10-fold cross validation for both models with DAAG package and was surprised to get identical (poor) accuracy results for both = 0.56

cv10<-CVbinary(model1,nfolds=10)

I thought that the model created using the dataset without outliers, having a much better fit, will give me higher accuracy. Am I missing something here? I will be grateful for your help.

$\endgroup$
2
$\begingroup$

Removal of "outliers" without outside information about data quality is not appropriate in my opinion. You didn't explain what it is about the "outliers" that makes you worry. It is a better approach to formulate a model that is likely to fit, e.g., allow continuous predictors to act in a smooth nonlinear fashion using regression splines (e.g., rms package rcs function) and to ask directed questions instead of emphasizing omnibus goodness of fit.

You are using an accuracy score that is a discontinuous improper score. Optimizing such a score leads one to choose the wrong variables, improperly transform them, and then to give them the wrong weights.

The link to which you referred us contains a lot of suggestions that reveal that the author does not run simulation studies to demonstrate the operating characteristics of the proposed methods.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.