I am coding in R, and I would like to verify that my multinomial logistic regression model is not overfit and to assess the performance of my model. The data that I am using is categorical and has many different variables (ie, age, cell count), but I selected only the ones I felt would be most relevant to the model.

I know that cross-validation is among the most optimal ways to accomplish this, but I am very new to this process. I poked around online for some examples, but a large portion of the resources I came across only explained cross-validation for models that used two variables. I did, however, learn that k-fold cross is a standard of cross validation--would it be applicable to a multinomial logistic regression model? What other methods are recommended?

Here are the lines of code that I have attempted:

k.folds <- function(k) {
      folds <- createFolds(PROJECTONE$categories, k = k, list = TRUE, returnTrain = TRUE)
      for (i in 1:k) {
        model <- multinom(categories ~ Age + Cell + Net + Per1 + Per2 + Per3 + Per4 + Lat + ML, 
                       data = PROJECTONE[folds[[i]],],
                       method = "class")
        predictions <- predict(object = model, newdata = PROJECTONE[-folds[[i]],], type = "class")
        accuracies.dt <- c(accuracies.dt, 
                           confusionMatrix(predictions, PROJECTONE[-folds[[i]], ]$categories)$overall[[1]])
accuracies.dt <- c()
accuracies.dt <- k.folds(5) 

mean.accuracies <- mean(accuracies.dt)

I received this:

[1] 0.7128463 0.7187894 0.7153652 0.7141058 0.7229219

as my result. I assume this is the confusion matrix that was created. Is there a way that I can better visualize these results, like plotting a graph? If so, what functions/lines of code should I use?

I have also tried repeated k-fold cross validation:

v <- c()
v <- replicate(200, k.folds(5))
accuracies.dt <- c()
for (i in 1 : 200) { 
  accuracies.dt <- c(accuracies.dt, v[,i])

mean.accuracies <- mean(accuracies.dt)
lci <- mean(accuracies.dt) - sd(accuracies.dt) * 1.96
uci <- mean(accuracies.dt) + sd(accuracies.dt) * 1.96

Both of these are examples I found off of the internet. However, I received a large response from the repeated k-fold cross validation (as in thousands of results generated). Is that normal? And same question as above--how can I better visualize the results for the model?

Are there any other ways to cross validate this multinomial logistic regression model? After I finish this process, I need to cross validate a boosting, random forest, decision tree, and many other models. Any suggestions on how to approach the rest of these as well?

Thank you for taking the time to help out a newbie. I know I asked a lot of questions, so if anything is unclear please tell me so.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.