I have a data frame wherein
60% of the values are missing I am replacing the missing values by
zero which is a legitimate zero which is making the data sparse. Now when I train the model with SVM for one class or novelty detection which is giving me good accuracy but I have another test datasets
for_prediction1 which has 100K records and sparse when I pass the
for_prediction1 to predict using the generated SVM model but it is predicting almost all of them as true which is not correct only few records should be true.
Is there any way to handle the sparse data so that it predicts the data accurately?
Can anyone help me in suggesting the pre-processing techniques that are available ?
smp_size <- floor(.9 * nrow(new_data)) train_ind <- sample(seq_len(nrow(new_data)), size = smp_size) train <- new_data[train_ind, ] test <- new_data[-train_ind, ] train_features <- train[,-ncol(train)] train_labels <- train[,ncol(train)] test_features <- test[,-ncol(test)] test_labels <- test[,ncol(test)] #,gamma = .001 svm.model <- svm(train$LABELS ~ ., data = train,nu =.001 ,gamma = .001,type = "one-classification" ,kernel= "radial",cross =10) summary(svm.model) Total Accuracy: 99.89021 Single Accuracies: 99.9617 99.85957 99.91063 99.9234 99.87233 99.8085 99.91063 99.91063 99.91063 99.83406 svm_pred1 <- predict(svm.model,test_features) conf_matrix <- table(pred = svm_pred1, true = t(test_labels)) conf_matrix true pred 1 FALSE 13 TRUE 8691 svm_pred <- predict(svm.model,for_prediction1) value count FALSE 6 TRUE 99994