1
$\begingroup$

I am looking to use a Decision Tree to classify whether or not a car will sell based on attributes of that car. The attributes that I have include price, year, mileage, condition (new, pre-owned, or used), number of cylinders (4, 6, 8), transmission_type (manual, auto, other). I have complete data for nearly 1,500 cars, of which 116 had sold.

I have followed many tutorials, and the process I am following is as such:

  • Randomly partition data into 70% train / 30% validation
  • Upsample training dataset to eliminate imblanace in selling status
  • Grow out complete decision tree with training data
  • Determine where to cut the decision tree based on minimum CP
  • Prune decision tree based on minimum CP
  • Fit pruned tree to test data
  • Evaluate fit based on confusion matrix

The problem that I'm experiencing is that the pruned tree really doesn't look much different from the complete tree. Also, the model doesn't do very well in correctly classifying observations in the minority group.

My question is whether my attempt at pruning is really doing anything? What am I missing in this process? I know there are other ways to fix the imbalanced training data, but I'm not sure if that's the problem, or if there is something else causing the issue.

If you are interested in looking at the data, I have made it available at he following URL: http://pastebin.com/qJkCmR6x

In addition, I have included my code below for your review. Please me know if if you have any thoughts on how I could improve the minority classification in this situation.

library(rpart)
library(caret)

# read CSV data into df
df <- read.csv("data.csv")

# set variables type accordingly
df$price <-  as.numeric(df$price)
df$year  <- as.ordered(factor(df$year))
df$condition <- factor(df$condition, levels=c("Used", "Certified pre-owned", "New"), ordered=TRUE)
df$numberofcylinders  <- as.ordered(factor(df$numberofcylinders))
df$transmission_type <- as.factor(df$transmission_type)
df$status <- as.factor(df$status)


## Create training and test data
# figure out 70% sample size
smp_size <- floor(0.70 * nrow(df))

# partition data into train and test
set.seed(123)
train_ind <- sample(seq_len(nrow(df)), size = smp_size)
train <- df[train_ind, ]
test <- df[-train_ind, ]

# upsample training data for equal proportions of 1 and 0
up_train <- upSample(x = train[, -ncol(train)],
                     y = train$status)

## Fit Decision Tree
# grow tree out completely
fit <-rpart(Class ~ price + year + mileage + condition + numberofcylinders + transmission_type,                         
            data = up_train,                   
            method = "class",                     
            parms = list(split = 'information'),
            maxsurrogate = 0,                     
            cp = 0,                              
            minsplit = 5,                                                             
            minbucket = 2,
            xval = 10)

# plot tree
plot(fit, uniform=TRUE, main="Decision Tree to Predict If Car Sold")
text(fit, use.n=TRUE, all=TRUE, cex=.8)

Full Decision Tree

# display the results
printcp(fit)

# detailied summany of splits
summary(fit)

# visualize cross validation results
plotcp(fit)

Size Of Tree

# determine where to cut the tree
fit$cptable[which.min(fit$cptable[,"xerror"]),"CP"]

# prune the tree to prevent overfitting
pfit<- prune(fit, cp = fit$cptable[which.min(fit$cptable[,"xerror"]),"CP"])


# show results of pruned tree
summary(pfit)

# plot pruned results
plot(fit, uniform=TRUE, main="Pruned Decision Tree to Predict If Car Sold")
text(fit, use.n=TRUE, all=TRUE, cex=.8)

Pruned Decision Tree

# fit pruned tree to test data
pred = predict(pfit, test, type="class")

# the breakdown of the actual status in the test data
table(test$status)

  0   1 
413  36 

# review predicted vs actual
table(pred, test$status)

pred   0   1
   0 392  30
   1  21   6

# calcaulate accuracy
sum(test$status==pred)/length(pred)

0.88641425389755
$\endgroup$
  • $\begingroup$ What does CP stand for? $\endgroup$ – Matthew Drury Feb 28 '17 at 23:48
  • $\begingroup$ When you calculated your misclassification percentage, what does that look like? You could try running this against the test data using both the pruned and unpruned data, and see if accuracy improves significantly, i.e. the lower the better. $\endgroup$ – Michael Grogan Feb 28 '17 at 23:57
  • $\begingroup$ @MatthewDrury -- I'm not very familiar with what exactly it does, but I see most examples use it, so I did as well. As seen in this documentation, stat.ethz.ch/R-manual/R-devel/library/rpart/html/…, CP stands for Complexity Parameter, and it means that any split that does not decrease the overall lack of fit by a factor of cp is not attempted. $\endgroup$ – CurtLH Feb 28 '17 at 23:59
  • $\begingroup$ Personally, I would use a random forest to fit it, then use it to generate many more and equally balanced samples, then fit a CART to that. Starting with a CART and only a single one is more challenging. A CART is a weak learner. $\endgroup$ – EngrStudent Mar 1 '17 at 0:02
  • 1
    $\begingroup$ @EngrStudent -- I agree that in the next step, it makes sense to move to Random Forest. However, I want to make sure that I'm correctly applying a single Decision Tree before I do that. If it looks like I've done things correctly up to this point, I have no problem exploring more complex methods. I just don't want to jump to that before I know I have the basics down first. :) $\endgroup$ – CurtLH Mar 1 '17 at 0:07
1
$\begingroup$

Old thread, but I think the "problem" is that your tree is very good, so pruning has no effect. Pruning means to eliminate leaves that are not increasing the accuracy significantly, i.e. to prevent overfitting. In your plotcp you are plotting the out-of-sample error of the model, computed by rpart through cross-validation: pruning simply removes leaves until you reach the minimum in that plot. The minimum is at cp=0, so there is nothing to prune.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.