0
$\begingroup$

I am using a dataset with a dicotomic class and testing how noise affects the decision tree j48 - from Rweka, using R - performance. I´m adding noise, and using confindence factors from 0.01 to 0.5 along the experiment.

As the class is dicotomic, it makes perfect sense that the worst performance of the training set is at 50%, being the class dicotomic.

Now... as the validation goes down, I´ve some points where the accuracy is almost perfect when the noise is total. How can I explain that?

Red is for the training set; Blue for the validation (80/20)

accuracy = (correctly predicted class / total testing class) × 100% OR, The percentage of correctly classified instances (TP + TN)/(TP + TN + FP + FN). where TP, FN, FP and TN represent the number of true positives, false negatives, false positives and true negatives, respectively.

The dataset I´m using is a random sample (10%) of this one https://www.kaggle.com/ludobenistant/hr-analytics

Here´s my code:

library(arules)
library(RWeka)
library(readr)
library(caret)
library(caTools)


setwd("C:\\Users\\Lucas\\Desktop\\AA")
inputFileName = "hr.csv"

ruido = function( datasetName, percentage ) {
  hr = read_csv(datasetName)

  sample = sample.split(hr$left, SplitRatio = percentage / 100)
  toModify = subset(hr, sample == TRUE)
  NotModify = subset(hr, sample == FALSE)

  toModify$left = sapply(toModify$left, function(x) !x )

  new_hr = rbind(toModify, toNotModify)

  output = strsplit(datasetName, "[.]")
  outputFileName = paste0(output[[1]][1], '_ruido_', percentage, '.', output[[1]][2])

  write.csv(new_hr, file = outputFileName, row.names=FALSE)
  outputFileName
}

set.seed(101) 
percentages = seq(0, 100 )
confidenceFactors = seq(0.05, 0.5, 0.05)

dataset = read.csv( inputFileName, sep = "," )

sample = sample.split(dataset$left, SplitRatio = .8)
train = subset(dataset, sample == TRUE)
test  = subset(dataset, sample == FALSE)

dir.create(file.path("C:\\Users\\Lucas\\Desktop\\AA", "datasets\\5"))
write.csv(train, file = "datasets\\5\\HR_train.csv", row.names=FALSE)
write.csv(test, file = "datasets\\5\\HR_test.csv", row.names=FALSE)

sizeResults = data.frame( "CF" = double(), "Percentage" = double(), "Value" = integer(), stringsAsFactors=FALSE)
accuracyResults = data.frame( "CF" = double(), "Method" = character(), "Percentage" = double(), "Value" = integer(), stringsAsFactors=FALSE)

for( percentage in percentages ){

    percentage = percentage / 100

    # load training dataset
    outputFile = ruido( "datasets\\5\\HR_train.csv", percentage * 100 )
    training = read.csv( outputFile )
    training$number_project = as.factor( training$number_project )
    training$time_spend_company = as.factor( training$time_spend_company )


    for( c in confidenceFactors ){
      # create tree
      print( paste0("Decision tree for: ", "percentage=", percentage, ", CF=", c) )
      tree = J48( as.factor(left) ~ ., training, control = Weka_control(M=2, C=c) )
      treeSize = tree$classifier$numElements()
      sizeResults = rbind(sizeResults, data.frame(CF=c, Percentage=percentage, Value=treeSize))

      # test tree
      pred <- predict(tree, test, type='class')
      trainingAccuracy = summary(tree)$details[[1]]
      testAccuracy = confusionMatrix(table(test$left, pred))
      accuracyResults = rbind(accuracyResults, data.frame(CF=c, Percentage=percentage, Method="Entrenamiento", Value=trainingAccuracy))
      accuracyResults = rbind(accuracyResults, data.frame(CF=c, Percentage=percentage, Method="Validaci�n", Value=testAccuracy$overall[[1]] * 100))
    }
}


ggplot(data=sizeResults, aes(x=Percentage, y=Value)) +
  geom_point() +
  ylab("Nodes") + # Set axis labels
  scale_colour_hue(name="")       # Set legend title  

ggplot(data=accuracyResults, aes(x=Percentage, y=Value, group=Method, color=Method)) +
  geom_point() +
  ylab("Accuracy") + # Set axis labels
  scale_colour_hue(name="") 
$\endgroup$
  • $\begingroup$ So what is the bottom axis? The percentage of the dataset that is represented by random noise?? $\endgroup$ – Josh Jun 14 '17 at 2:54
  • $\begingroup$ @Josh it´s the noise % in the class. The class is dicotomic, so a 100% noise means all the class labels got inverted. $\endgroup$ – Luis Jun 14 '17 at 3:00
  • $\begingroup$ I see... And now I see you're referring to that single blue dot with the arrow. Wow! $\endgroup$ – Josh Jun 14 '17 at 3:10
  • $\begingroup$ Yes. Do you know what that may be? $\endgroup$ – Luis Jun 14 '17 at 3:12
  • $\begingroup$ no clue! This feels like more of a coding question and might do better in stack overflow $\endgroup$ – Josh Jun 14 '17 at 3:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.