I am looking to use a Decision Tree to classify whether or not a car will sell based on attributes of that car. The attributes that I have include price, year, mileage, condition (new, pre-owned, or used), number of cylinders (4, 6, 8), transmission_type (manual, auto, other). I have complete data for nearly 1,500 cars, of which 116 had sold.
I have followed many tutorials, and the process I am following is as such:
- Randomly partition data into 70% train / 30% validation
- Upsample training dataset to eliminate imblanace in selling status
- Grow out complete decision tree with training data
- Determine where to cut the decision tree based on minimum CP
- Prune decision tree based on minimum CP
- Fit pruned tree to test data
- Evaluate fit based on confusion matrix
The problem that I'm experiencing is that the pruned tree really doesn't look much different from the complete tree. Also, the model doesn't do very well in correctly classifying observations in the minority group.
My question is whether my attempt at pruning is really doing anything? What am I missing in this process? I know there are other ways to fix the imbalanced training data, but I'm not sure if that's the problem, or if there is something else causing the issue.
If you are interested in looking at the data, I have made it available at he following URL: http://pastebin.com/qJkCmR6x
In addition, I have included my code below for your review. Please me know if if you have any thoughts on how I could improve the minority classification in this situation.
library(rpart) library(caret) # read CSV data into df df <- read.csv("data.csv") # set variables type accordingly df$price <- as.numeric(df$price) df$year <- as.ordered(factor(df$year)) df$condition <- factor(df$condition, levels=c("Used", "Certified pre-owned", "New"), ordered=TRUE) df$numberofcylinders <- as.ordered(factor(df$numberofcylinders)) df$transmission_type <- as.factor(df$transmission_type) df$status <- as.factor(df$status) ## Create training and test data # figure out 70% sample size smp_size <- floor(0.70 * nrow(df)) # partition data into train and test set.seed(123) train_ind <- sample(seq_len(nrow(df)), size = smp_size) train <- df[train_ind, ] test <- df[-train_ind, ] # upsample training data for equal proportions of 1 and 0 up_train <- upSample(x = train[, -ncol(train)], y = train$status) ## Fit Decision Tree # grow tree out completely fit <-rpart(Class ~ price + year + mileage + condition + numberofcylinders + transmission_type, data = up_train, method = "class", parms = list(split = 'information'), maxsurrogate = 0, cp = 0, minsplit = 5, minbucket = 2, xval = 10) # plot tree plot(fit, uniform=TRUE, main="Decision Tree to Predict If Car Sold") text(fit, use.n=TRUE, all=TRUE, cex=.8)
# display the results printcp(fit) # detailied summany of splits summary(fit) # visualize cross validation results plotcp(fit)
# determine where to cut the tree fit$cptable[which.min(fit$cptable[,"xerror"]),"CP"] # prune the tree to prevent overfitting pfit<- prune(fit, cp = fit$cptable[which.min(fit$cptable[,"xerror"]),"CP"]) # show results of pruned tree summary(pfit) # plot pruned results plot(fit, uniform=TRUE, main="Pruned Decision Tree to Predict If Car Sold") text(fit, use.n=TRUE, all=TRUE, cex=.8)
# fit pruned tree to test data pred = predict(pfit, test, type="class") # the breakdown of the actual status in the test data table(test$status) 0 1 413 36 # review predicted vs actual table(pred, test$status) pred 0 1 0 392 30 1 21 6 # calcaulate accuracy sum(test$status==pred)/length(pred) 0.88641425389755