0
$\begingroup$

I got a data set from a hospital from the PET/CT lab on oncology data (which I'm very unfortunately not allowed to share).

n is 126 with 116 features.

There is one target variable called death which I have to predict.

I've written two small ApacheSpark applications in Scala - one using SVM w/ SGD and one using Decision Trees. On SVM I get an ROC of 0.5 and on Decision Trees I get 0.42 error rate.

My question is if there is a chance with that small number of examples do to any useful prediction and if so if you can give me some advice on how to proceed.

Below there is my code for Decision Trees:

import org.apache.spark.mllib.linalg.{Vector, Vectors}
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.classification.{SVMModel, SVMWithSGD}
import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
import org.apache.spark.mllib.tree.DecisionTree
import org.apache.spark.mllib.tree.model.DecisionTreeModel
import org.apache.spark.mllib.util.MLUtils

var rdd = sc.textFile("/Users/romeokienzler/Documents/proj/eoc/Workbook1.csv")

//convert date to number
//convert % to number
//convert . to zero
//convert csv to array
//convert empty strings to 0
var matrix = rdd.map(s => s.replaceAll("/","")).map(s => s.replaceAll("%","")).map(s => s.replace('.','0')).map(s => s.split(',')).map(a => a.map(e => if (e.length==0) "0" else e))

//convert double matrix to mllib matrix
var matrixMLLib = matrix.map(a => a.map(e => e.toDouble)).filter(a=> a.length==89).map(a => new LabeledPoint(a(1),Vectors.dense((a.slice(2,a.length)))))

// Split data into training (60%) and test (40%).
val splits = matrixMLLib.randomSplit(Array(0.6, 0.4), seed = 11L)
val (trainingData, testData) = (splits(0), splits(1))

// Train a DecisionTree model.
//  Empty categoricalFeaturesInfo indicates all features are continuous.
val numClasses = 2
val categoricalFeaturesInfo = Map[Int, Int]()
val impurity = "gini"
val maxDepth = 5
val maxBins = 32

val model = DecisionTree.trainClassifier(trainingData, numClasses, categoricalFeaturesInfo,
  impurity, maxDepth, maxBins)

// Evaluate model on test instances and compute test error
val labelAndPreds = testData.map { point =>
  val prediction = model.predict(point.features)
  (point.label, prediction)
}
val testErr = labelAndPreds.filter(r => r._1 != r._2).count.toDouble / testData.count()
println("Test Error = " + testErr)
println("Learned classification tree model:\n" + model.toDebugString)

// Save and load model
model.save(sc, "myModelPath")
val sameModel = DecisionTreeModel.load(sc, "myModelPath")

Support Vector Machine:

import org.apache.spark.mllib.linalg.{Vector, Vectors}
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.classification.{SVMModel, SVMWithSGD}
import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics

var rdd = sc.textFile("/user/root/romeo/eoc/data.csv")

//convert date to number
//convert % to number
//convert . to zero
//convert csv to array
//convert empty strings to 0
var matrix = rdd.map(s => s.replaceAll("/","")).map(s => s.replaceAll("%","")).map(s => s.replace('.','0')).map(s => s.split(',')).map(a => a.map(e => if (e.length==0) "0" else e))

//convert double matrix to mllib matrix
var matrixMLLib = matrix.map(a => a.map(e => e.toDouble)).filter(a=> a.length==89).map(a => new LabeledPoint(a(1),Vectors.dense((a.slice(2,a.length)))))

// Split data into training (60%) and test (40%).
val splits = matrixMLLib.randomSplit(Array(0.9, 0.1), seed = 11L)
val training = splits(0)
val test = splits(1)

val numIterations = 1000
val model = SVMWithSGD.train(training, numIterations)

// Compute raw scores on the test set.
val scoreAndLabels = test.map { point =>
  val score = model.predict(point.features)
  (score, point.label)
}

// Get evaluation metrics.
val metrics = new BinaryClassificationMetrics(scoreAndLabels)
val auROC = metrics.areaUnderROC()

println("Area under ROC = " + auROC)

Thanks a lot in advance!

$\endgroup$
1
  • $\begingroup$ Not directly related to the question, but why bother with Spark for such small datasets? You'd be better off using the wealth of algorithms available in packages like scikit-learn. $\endgroup$ Nov 2 '15 at 12:43
2
$\begingroup$

Romeo, for such small data sets it can be hard to build good classifiers while avoiding overfitting. Also the domain you mentioned often deals with poor quality data / lots of noise.

My first guess would be that you simply run into the "curse of dimensionality" (with 116 features vs. 126 data points): https://en.wikipedia.org/wiki/Curse_of_dimensionality (in high dimensional data spaces many ML algorithms and distance metrics (e.g. euclidean distance) perform poorly.)

So you might not get there by simply tweaking one particular algorithm but a.) look at preprocessing/dimension reduction and b.) ensemble methods.

Things you could try:

  • Try some feature selection/dimensionality reduction, e.g. PCA
  • Think about data cleansing. Can you identify outliers? are these obviously wrong data entries, or maybe actually the info you're looking for?
  • Use ensemble methods (like stacked generalizations, boosting/bagging) to build different models, then build a meta-classifier

Maybe that helps. - Florian

$\endgroup$
1
  • $\begingroup$ Dear Florian, this actually helped, I've created a RandomForrest Classifier searching in multiple nested loops through the parameter space and ranked the parameters based on test error. Finally I've evaluated the chosen parameter-set using the validation partition and now I get a validation error of 5% - I'm happy! :) $\endgroup$ Nov 18 '15 at 23:42
1
$\begingroup$

A few issues straight off the bat related to your SVM code:

  • you are using linear SVM, which probably won't perform very well given the low dimensionality (but better than random)

  • you did not set the misclassification penalty C ($\approx$ regparam in Spark), so I guess you're using the default value (whatever that may be). This parameter needs to be optimized by you.

  • you are usingn 1000 SGD iterations, which is probably not enough for the SVM to converge. If you use more appropriate software, you wouldn't have to deal with these kinds of parameters, which only really become relevant for huge data sets.

As I mentioned in the comment, you should consider more suitable libraries for such small data sets.

$\endgroup$
0
$\begingroup$

Just for the sake of completeness I'm posting my working solution here (just the raw prototype, nothing beautified so far)

import org.apache.spark.mllib.linalg.{ Vector, Vectors }
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.classification.{ SVMModel, SVMWithSGD }
import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
import org.apache.spark.mllib.tree.RandomForest
import org.apache.spark.mllib.util.MLUtils


  var rdd = sc.textFile("/Users/romeokienzler/Documents/proj/eoc/ielsg26ALLsept_2015_IBM.csv")
  rdd.cache()
  val headerAndRows = rdd.map(line => line.split(",").map(_.trim))
  val header = headerAndRows.first
  val data = headerAndRows.filter(_(0) != header(0))
  //val maps = data.map(splits => header.zip(splits))O
  //val result = maps.filter(map => map("user") != "me")

  //convert date to number
  //convert % to number
  //convert . to zero
  //convert csv to array
  //convert empty strings to 0
  //var matrix = rdd.map(s => s.replaceAll("/","")).map(s => //s.replaceAll("%","")).map(s => s.replace('.','0')).map(s => s.split(',')).map(a //=> a.map(e => if (e.length==0) "0" else e))

  var matrix = data.map(a => a.map(s => s.replaceAll("/", "")).map(s => s.replaceAll("%", "")).map(s => s.replace('.', '0')).map(s => s.split(',')).map(a => a.map(e => if (e.length == 0) "0" else e)))

  //var matrix = data.map(s => s.replaceAll("/","")).map(s => //s.replaceAll("%","")).map(s => s.replace('.','0')).map(s => s.split(',')).map(a //=> a.map(e => if (e.length==0) "0" else e))

  //convert double matrix to mllib matrix
  var matrixMLLib = matrix.map(a => a.map(aa => aa(0))).filter(a => a.length == 90).map(a => a.map(e => e.toDouble)).map(a => new LabeledPoint(a(83), Vectors.dense((a.slice(0, 83) ++ a.slice(84, a.length)))))

  //var matrixMLLib = matrix.map(a => a.map(e => e.toDouble)).filter(a=> //a.length==89).map(a => new //LabeledPoint(a(1),Vectors.dense((a.slice(2,a.length)))))

  val splits = matrixMLLib.randomSplit(Array(0.6, 0.2,0.2))
  val (trainingData, testData, validationData) = (splits(0), splits(1),splits(2))

  // Train a RandomForest model.
  //  Empty categoricalFeaturesInfo indicates all features are continuous.
  val numClassesList = 2 :: 3 :: 4 :: 5 :: 6 :: 7 :: 8 :: 9 :: 10 :: Nil
  val categoricalFeaturesInfo = Map[Int, Int]()
  val numTreesList = 3 :: 5 :: 10 :: 30 :: 50 :: 70 :: 100 :: 150 :: Nil // Use more in practice.
  val featureSubsetStrategyList = "auto" :: "all" :: "sqrt" :: "log2" :: "onethird" :: Nil // Let the algorithm choose.
  val impurityList = "gini" :: "entropy" :: Nil
  val maxDepthList = 3 :: 5 :: 10 :: 15 :: 20 :: 25 :: 30 :: Nil
  val maxBinsList = 2 :: 5 :: 10 :: 15 :: 20 :: 25 :: 30 :: Nil

  var parametersAndScore: List[(Double, Int, Int, String, String, Int, Int)] = Nil

  for (numTrees <- numTreesList) {
    for (maxDepth <- maxDepthList) {
      for (maxBins <- maxBinsList) {
        for (impurity <- impurityList) {
          for (featureSubsetStrategy <- featureSubsetStrategyList) {
            for (numClasses <- numClassesList) {
              val model = RandomForest.trainClassifier(trainingData, numClasses, categoricalFeaturesInfo,
                numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins)

              // Evaluate model on test instances and compute test error
              val labelAndPreds = testData.map { point =>
                val prediction = model.predict(point.features)
                (point.label, prediction)
              }
              val testErr = labelAndPreds.filter(r => r._1 != r._2).count.toDouble / testData.count()
              println("Test Error = " + (testErr, numClasses,
                numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins))
              parametersAndScore = (testErr, numClasses,
                numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins) :: parametersAndScore
            }
          }
        }
      }
    }
  }
val optimalParameters =  parametersAndScore.sortBy(_._1).sortBy(_._3).head

//(0.0,3,10,auto,gini,3,2)
val model = RandomForest.trainClassifier(trainingData, optimalParameters._2, categoricalFeaturesInfo ,optimalParameters._3, optimalParameters._4, optimalParameters._5,optimalParameters._6,optimalParameters._7)
// Evaluate model on test instances and compute test error
              val labelAndPreds = validationData.map { point =>
                val prediction = model.predict(point.features)
                (point.label, prediction)
              }
              val validationErr = labelAndPreds.filter(r => r._1 != r._2).count.toDouble / validationData.count()
              println("Validation Error = " + validationErr)
println("Learned classification forest model:\n" + model.toDebugString)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.