# Poor multiclass classification using Caret in R [closed]

I have 5 continuous variables with 211 rows of data and each row is assigned a region, there are 7 possible regions in total.

I wish to build a machine learning model that classifies an unseen row of 5 numerical variables and classifies it as one of the 7 regions.

I have tried building a decision tree classifier as well as a Naive Bayes classifier but the accuracy for both models are really poor, 53% for both models.

I believe I have configured both models correctly but I cannot improve the accuracy. In summary is this telling me that both model are not appropriate for the task at hand (multiclass classification)?

For a reproducible example, here is a link to a dput of my variables and below is the code I am using for both models.

#data preparation for decision tree classification
x <- nationsCombImputed[,c(5,6,7,8,11)]
y <- nationsCombImputed[,9]
#numericalVars$region <- relevel(numericalVars$region, ref = "South Asia")
exampleData <- c(55, 2000000, 40, 30, 1500)
exampleData <- as.data.frame(matrix(exampleData,nrow = 1))
colnames(exampleData) <- c("life_expect",
"population",
"birth_rate",
"neonat_mortal_rate",
"gdp_percap.x")

# define training control using k-fold Cross Validation
train_control <- trainControl(method="cv", number=10)

preProcessInTrain <-c("center", "scale")

#train the regression tree model
rtModel <- train(x,y,
trControl=train_control,
method="rpart",
tuneLength = 10,
parms = list(split="information"),
preProc = preProcessInTrain)

# train the naive bayes model
nbModel <- train(x,y,
trControl=train_control,
method="nb",
preProc = preProcessInTrain)

print(rtModel)
print(nbModel)


## closed as off-topic by Michael Chernick, Peter Flom♦Oct 31 '17 at 10:53

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

I think that the problem is not that you are using the classification methods poorly, but rather that this data has little predictive power for the regions.

First of all, two classes have little data.

table(y)
East Asia & Pacific
35
Europe & Central Asia
56
Latin America & Caribbean
41
Middle East & North Africa
21
North America
3
South Asia
8
Sub-Saharan Africa
47


North America has only three points to learn from and South Asia has only 8. That is not a lot of data from which to derive distinguishing statistics. And indeed, your decision tree gets all of these points wrong, as we will see soon. Well, that is only 11 points out of 211 (5%), so if the rest are all right the accuracy can still be OK.

Now let's look at the variables. They are pretty highly correlated

cor(x)
life_expect   population  birth_rate neonat_mortal_rate gdp_percap.x
life_expect         1.000000000  0.001938783 -0.86925842        -0.86918279    0.6311007
population          0.001938783  1.000000000 -0.04339276         0.06374319   -0.0695579
birth_rate         -0.869258422 -0.043392764  1.00000000         0.83330008   -0.5893538
neonat_mortal_rate -0.869182785  0.063743194  0.83330008         1.00000000   -0.6033750
gdp_percap.x        0.631100741 -0.069557904 -0.58935378        -0.60337498    1.0000000


In fact, looking at principal components, it looks like we really only have one dimension of data.

plot(princomp(x))


How well do the individual variables distinguish the regions? Taking a look at the distributions using sm.density.compare gives some idea. Here is life_expect

library(sm)
sm.density.compare(x[,1], y, col=rainbow(7))
legend("topleft", levels(y), lty=1, col=rainbow(7))


You can see that Sub-Saharan Africa is a bit distinctive, but the other regions overlap each other so much that we have little hope of distinguishing them. Looking at the other variables, population is extreme in its lack of information. (China and India are the extreme outliers.)

Step through all 5 predictor variables and you will see that most classes (regions) will be difficult to distinguish.

But we don't just use one variable by itself; we use them in combination. So let's go to 2-dimensions. Perhaps the best of the scatterplots is life_expect vs birth_rate.

plot(x[,1], x[,3], pch=20, col=rainbow(7)[y])
legend("topright", levels(y), pch=20, col=rainbow(7))


You can see that Sub-Saharan Africa looks pretty well separated from the other regions. Further, we can hope for some separation of Europe & Central Asia. East Asia & Pacific is all over the place. The bad color map makes it hard to tell about Middle East & North Africa vs Latin America & Caribbean.

Now let's look at what really happened. Here is the confusion matrix.

table(predict(rtModel), y)
y
East Asia & Pacific Europe & Central Asia Latin America & Caribbean
East Asia & Pacific                          0                     0                         0
Europe & Central Asia                       11                    47                        13
Latin America & Caribbean                   17                     8                        25
Middle East & North Africa                   0                     0                         0
North America                                0                     0                         0
South Asia                                   0                     0                         0
Sub-Saharan Africa                           7                     1                         3
y
Middle East & North Africa North America South Asia Sub-Saharan Africa
East Asia & Pacific                                 0             0          0                  0
Europe & Central Asia                               3             3          0                  1
Latin America & Caribbean                          16             0          6                  1
Middle East & North Africa                          0             0          0                  0
North America                                       0             0          0                  0
South Asia                                          0             0          0                  0
Sub-Saharan Africa                                  2             0          2                 45


The diagonal gives the correctly classified countries. Notice that NONE of the examples from East Asia & Pacific, Middle East & North Africa, North America, or South Asia were correctly classsified. Latin America & Caribbean is only about 60% correct. Middle East & North Africa got 16/21 (76%) correct. We did pretty well on Sub-Saharan Africa (96%) and Europe & Central Asia (84%). Notice that the actual results agree pretty well with expectations based on looking at the scatter plots.

In summary, the real problem is the data for prediction. The variables are highly correlated. The distributions for the regions mostly overlap each other. We have only a few examples of some classes. You should not have high expectations for predictions from this data.

• thank you so much for your detailed reply. Although I generated some of the same plots as you did I wasn't able to tie the "story" together as succinctly as you have. Many thanks once again. – TheGoat Oct 31 '17 at 14:18