0
$\begingroup$

I'm trying to build a specific neural network architecture and testing it using 10 fold cross validation of a dataset. Now building the model is a tedious job and Weka expects me to make it 10 times for each of the 10 folds.

Can't I just make the model for the first fold and ask weka to use that same model for the remaining 9?

Also, since building neural networks in weka is so easy, can I import a neural network structure to matlab for further use?

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ 10-fold CV consists of building the model 10-times. There is no way around that. After all, in each fold you will have different train set. $\endgroup$
    – alesc
    Apr 25, 2015 at 19:43
  • 2
    $\begingroup$ I think you need to read up on what cross-validation does. Based on your question, it appears you don't know what cross-validation is for. $\endgroup$
    – Marc Claesen
    Aug 27, 2015 at 18:08
  • $\begingroup$ Chapter 5 of An Introduction to Statistical Learning provides a good intro to cross validation. $\endgroup$
    – Tchotchke
    Aug 27, 2015 at 19:24
  • $\begingroup$ Could you tell us why you would want to use multi fold cross validation instead of a simpler train test split (1 fold). Since neural networks take longer to train, train test splits are used more often than n fold validation. $\endgroup$
    – shark8me
    Apr 27, 2016 at 7:31

1 Answer 1

Reset to default
1
$\begingroup$

You can use the Evaluate class to perform this 10-fold cross-validation. To define the cross-validation you have to set the parameter as '-x 10' in the EvaluateModel.

clear all; close all; clc;

%% Add jar file to path plus import dependencies
javaaddpath('/usr/local/weka-3-6-11/weka.jar');
import weka.classifiers.trees.RandomForest.*;
import weka.classifiers.meta.Bagging.*;
import weka.classifiers.Evaluation.*;
import weka.core.Instances.*

%% load the arff file and extract the informations
filename = 'algo_output/results_features_labeling2_2class.arff';
loader = weka.core.converters.ArffLoader();
loader.setFile(java.io.File(filename));
data = loader.getDataSet();
data.setClassIndex(D.numAttributes()-1);

%% classification
classifier = weka.classifiers.functions.MultilayerPerceptron();
classifier.buildClassifier(data);
classifier.toString()

%% 10-fold cross-validation
ev = weka.classifiers.Evaluation(data);

v(1) = java.lang.String('-t');
v(2) = java.lang.String(filename);
v(3) = java.lang.String('-x');
v(4) = java.lang.String('10');
v(5) = java.lang.String('-i');

prm = cat(1,v(1:end));
ev.evaluateModel(classifier, prm)

For more information, check this link.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Just a remark: Weka's internal algorithm for performing $k$-fold CV is single-threaded. Since you are dealing with Neural Networks, which take a very long to train, you will desperately need multi-threading. If you need a Java implementation of a multi-threaded $k$-fold CV, I can provide you the code. $\endgroup$
    – alesc
    Apr 25, 2015 at 19:47
  • $\begingroup$ Yeah sure, I would be more than happy to see your code (even tho my classification run quite fast for my -really- small data-set). $\endgroup$
    – R.Falque
    Apr 26, 2015 at 0:19
  • $\begingroup$ If you have small samples, then even MultilayerPerceptron will finish quickly. I have been using large datasets that can take days to train, and without multi-threading, it would take much longer. Here is my code for benchmarking: pastebin. I have listed the source code for 3 files, but the first file is the most important. Since I am using Streams and lambda expressions, you will need Java 8 to run it. $\endgroup$
    – alesc
    Apr 26, 2015 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.