0
$\begingroup$

I'm trying to build a specific neural network architecture and testing it using 10 fold cross validation of a dataset. Now building the model is a tedious job and Weka expects me to make it 10 times for each of the 10 folds.

Can't I just make the model for the first fold and ask weka to use that same model for the remaining 9?

Also, since building neural networks in weka is so easy, can I import a neural network structure to matlab for further use?

$\endgroup$
  • 3
    $\begingroup$ 10-fold CV consists of building the model 10-times. There is no way around that. After all, in each fold you will have different train set. $\endgroup$ – alesc Apr 25 '15 at 19:43
  • 2
    $\begingroup$ I think you need to read up on what cross-validation does. Based on your question, it appears you don't know what cross-validation is for. $\endgroup$ – Marc Claesen Aug 27 '15 at 18:08
  • $\begingroup$ Chapter 5 of An Introduction to Statistical Learning provides a good intro to cross validation. $\endgroup$ – Tchotchke Aug 27 '15 at 19:24
  • $\begingroup$ Could you tell us why you would want to use multi fold cross validation instead of a simpler train test split (1 fold). Since neural networks take longer to train, train test splits are used more often than n fold validation. $\endgroup$ – shark8me Apr 27 '16 at 7:31
1
$\begingroup$

You can use the Evaluate class to perform this 10-fold cross-validation. To define the cross-validation you have to set the parameter as '-x 10' in the EvaluateModel.

clear all; close all; clc;

%% Add jar file to path plus import dependencies
javaaddpath('/usr/local/weka-3-6-11/weka.jar');
import weka.classifiers.trees.RandomForest.*;
import weka.classifiers.meta.Bagging.*;
import weka.classifiers.Evaluation.*;
import weka.core.Instances.*

%% load the arff file and extract the informations
filename = 'algo_output/results_features_labeling2_2class.arff';
loader = weka.core.converters.ArffLoader();
loader.setFile(java.io.File(filename));
data = loader.getDataSet();
data.setClassIndex(D.numAttributes()-1);

%% classification
classifier = weka.classifiers.functions.MultilayerPerceptron();
classifier.buildClassifier(data);
classifier.toString()

%% 10-fold cross-validation
ev = weka.classifiers.Evaluation(data);

v(1) = java.lang.String('-t');
v(2) = java.lang.String(filename);
v(3) = java.lang.String('-x');
v(4) = java.lang.String('10');
v(5) = java.lang.String('-i');

prm = cat(1,v(1:end));
ev.evaluateModel(classifier, prm)

For more information, check this link.

$\endgroup$
  • $\begingroup$ Just a remark: Weka's internal algorithm for performing $k$-fold CV is single-threaded. Since you are dealing with Neural Networks, which take a very long to train, you will desperately need multi-threading. If you need a Java implementation of a multi-threaded $k$-fold CV, I can provide you the code. $\endgroup$ – alesc Apr 25 '15 at 19:47
  • $\begingroup$ Yeah sure, I would be more than happy to see your code (even tho my classification run quite fast for my -really- small data-set). $\endgroup$ – R.Falque Apr 26 '15 at 0:19
  • $\begingroup$ If you have small samples, then even MultilayerPerceptron will finish quickly. I have been using large datasets that can take days to train, and without multi-threading, it would take much longer. Here is my code for benchmarking: pastebin. I have listed the source code for 3 files, but the first file is the most important. Since I am using Streams and lambda expressions, you will need Java 8 to run it. $\endgroup$ – alesc Apr 26 '15 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.