2
$\begingroup$

I have 5 continuous variables with 211 rows of data and each row is assigned a region, there are 7 possible regions in total.

I wish to build a machine learning model that classifies an unseen row of 5 numerical variables and classifies it as one of the 7 regions.

I have tried building a decision tree classifier as well as a Naive Bayes classifier but the accuracy for both models are really poor, 53% for both models.

I believe I have configured both models correctly but I cannot improve the accuracy. In summary is this telling me that both model are not appropriate for the task at hand (multiclass classification)?

For a reproducible example, here is a link to a dput of my variables and below is the code I am using for both models.

#data preparation for decision tree classification
x <- nationsCombImputed[,c(5,6,7,8,11)]
y <- nationsCombImputed[,9]
#numericalVars$region <- relevel(numericalVars$region, ref = "South Asia")
exampleData <- c(55, 2000000, 40, 30, 1500)
exampleData <- as.data.frame(matrix(exampleData,nrow = 1))
colnames(exampleData) <- c("life_expect", 
                           "population", 
                           "birth_rate", 
                           "neonat_mortal_rate", 
                           "gdp_percap.x")

# define training control using k-fold Cross Validation
train_control <- trainControl(method="cv", number=10)

preProcessInTrain <-c("center", "scale")

#train the regression tree model
rtModel <- train(x,y,
                 trControl=train_control,
                 method="rpart",
                 tuneLength = 10,
                 parms = list(split="information"),
                 preProc = preProcessInTrain)

# train the naive bayes model
nbModel <- train(x,y,
                 trControl=train_control, 
                 method="nb",
                 preProc = preProcessInTrain)

print(rtModel)
print(nbModel)
$\endgroup$

closed as off-topic by Michael Chernick, Peter Flom Oct 31 '17 at 10:53

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

6
$\begingroup$

I think that the problem is not that you are using the classification methods poorly, but rather that this data has little predictive power for the regions.

First of all, two classes have little data.

table(y)
       East Asia & Pacific 
                        35 
     Europe & Central Asia 
                        56 
 Latin America & Caribbean 
                        41 
Middle East & North Africa 
                        21 
             North America 
                         3 
                South Asia 
                         8 
        Sub-Saharan Africa 
                        47

North America has only three points to learn from and South Asia has only 8. That is not a lot of data from which to derive distinguishing statistics. And indeed, your decision tree gets all of these points wrong, as we will see soon. Well, that is only 11 points out of 211 (5%), so if the rest are all right the accuracy can still be OK.

Now let's look at the variables. They are pretty highly correlated

cor(x)
                    life_expect   population  birth_rate neonat_mortal_rate gdp_percap.x
life_expect         1.000000000  0.001938783 -0.86925842        -0.86918279    0.6311007
population          0.001938783  1.000000000 -0.04339276         0.06374319   -0.0695579
birth_rate         -0.869258422 -0.043392764  1.00000000         0.83330008   -0.5893538
neonat_mortal_rate -0.869182785  0.063743194  0.83330008         1.00000000   -0.6033750
gdp_percap.x        0.631100741 -0.069557904 -0.58935378        -0.60337498    1.0000000

In fact, looking at principal components, it looks like we really only have one dimension of data.

plot(princomp(x))

Principal Components

How well do the individual variables distinguish the regions? Taking a look at the distributions using sm.density.compare gives some idea. Here is life_expect

library(sm)
sm.density.compare(x[,1], y, col=rainbow(7))
legend("topleft", levels(y), lty=1, col=rainbow(7))

Density plot for life_expect

You can see that Sub-Saharan Africa is a bit distinctive, but the other regions overlap each other so much that we have little hope of distinguishing them. Looking at the other variables, population is extreme in its lack of information. (China and India are the extreme outliers.)

Population

Step through all 5 predictor variables and you will see that most classes (regions) will be difficult to distinguish.

But we don't just use one variable by itself; we use them in combination. So let's go to 2-dimensions. Perhaps the best of the scatterplots is life_expect vs birth_rate.

plot(x[,1], x[,3], pch=20, col=rainbow(7)[y])
legend("topright", levels(y), pch=20, col=rainbow(7))

life_expect vs birth_rate

You can see that Sub-Saharan Africa looks pretty well separated from the other regions. Further, we can hope for some separation of Europe & Central Asia. East Asia & Pacific is all over the place. The bad color map makes it hard to tell about Middle East & North Africa vs Latin America & Caribbean.

Now let's look at what really happened. Here is the confusion matrix.

table(predict(rtModel), y)
                            y
                             East Asia & Pacific Europe & Central Asia Latin America & Caribbean
  East Asia & Pacific                          0                     0                         0
  Europe & Central Asia                       11                    47                        13
  Latin America & Caribbean                   17                     8                        25
  Middle East & North Africa                   0                     0                         0
  North America                                0                     0                         0
  South Asia                                   0                     0                         0
  Sub-Saharan Africa                           7                     1                         3
                            y
                             Middle East & North Africa North America South Asia Sub-Saharan Africa
  East Asia & Pacific                                 0             0          0                  0
  Europe & Central Asia                               3             3          0                  1
  Latin America & Caribbean                          16             0          6                  1
  Middle East & North Africa                          0             0          0                  0
  North America                                       0             0          0                  0
  South Asia                                          0             0          0                  0
  Sub-Saharan Africa                                  2             0          2                 45

The diagonal gives the correctly classified countries. Notice that NONE of the examples from East Asia & Pacific, Middle East & North Africa, North America, or South Asia were correctly classsified. Latin America & Caribbean is only about 60% correct. Middle East & North Africa got 16/21 (76%) correct. We did pretty well on Sub-Saharan Africa (96%) and Europe & Central Asia (84%). Notice that the actual results agree pretty well with expectations based on looking at the scatter plots.

In summary, the real problem is the data for prediction. The variables are highly correlated. The distributions for the regions mostly overlap each other. We have only a few examples of some classes. You should not have high expectations for predictions from this data.

$\endgroup$
  • $\begingroup$ thank you so much for your detailed reply. Although I generated some of the same plots as you did I wasn't able to tie the "story" together as succinctly as you have. Many thanks once again. $\endgroup$ – TheGoat Oct 31 '17 at 14:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.