1
$\begingroup$

I am coding in R, and I would like to verify that my multinomial logistic regression model is not overfit and to assess the performance of my model. The data that I am using is categorical and has many different variables (ie, age, cell count), but I selected only the ones I felt would be most relevant to the model.

I know that cross-validation is among the most optimal ways to accomplish this, but I am very new to this process. I poked around online for some examples, but a large portion of the resources I came across only explained cross-validation for models that used two variables. I did, however, learn that k-fold cross is a standard of cross validation--would it be applicable to a multinomial logistic regression model? What other methods are recommended?

Here are the lines of code that I have attempted:

k.folds <- function(k) {
      folds <- createFolds(PROJECTONE$categories, k = k, list = TRUE, returnTrain = TRUE)
      for (i in 1:k) {
        model <- multinom(categories ~ Age + Cell + Net + Per1 + Per2 + Per3 + Per4 + Lat + ML, 
                       data = PROJECTONE[folds[[i]],],
                       method = "class")
        predictions <- predict(object = model, newdata = PROJECTONE[-folds[[i]],], type = "class")
        accuracies.dt <- c(accuracies.dt, 
                           confusionMatrix(predictions, PROJECTONE[-folds[[i]], ]$categories)$overall[[1]])
      }
      accuracies.dt
    }
set.seed(500)
accuracies.dt <- c()
accuracies.dt <- k.folds(5) 
accuracies.dt

mean.accuracies <- mean(accuracies.dt)

I received this:

[1] 0.7128463 0.7187894 0.7153652 0.7141058 0.7229219

as my result. I assume this is the confusion matrix that was created. Is there a way that I can better visualize these results, like plotting a graph? If so, what functions/lines of code should I use?

I have also tried repeated k-fold cross validation:

set.seed(500)
v <- c()
v <- replicate(200, k.folds(5))
accuracies.dt <- c()
for (i in 1 : 200) { 
  accuracies.dt <- c(accuracies.dt, v[,i])
}

mean.accuracies <- mean(accuracies.dt)
lci <- mean(accuracies.dt) - sd(accuracies.dt) * 1.96
uci <- mean(accuracies.dt) + sd(accuracies.dt) * 1.96

Both of these are examples I found off of the internet. However, I received a large response from the repeated k-fold cross validation (as in thousands of results generated). Is that normal? And same question as above--how can I better visualize the results for the model?

Are there any other ways to cross validate this multinomial logistic regression model? After I finish this process, I need to cross validate a boosting, random forest, decision tree, and many other models. Any suggestions on how to approach the rest of these as well?

Thank you for taking the time to help out a newbie. I know I asked a lot of questions, so if anything is unclear please tell me so.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.