Skip to main content
Cross Validated

Return to Question

Clarified original run, added details on new runs
Source Link
treed
treed
  • 103
  • 3

I am attempting to use a neural network, after using other machine learning algorithms. I am using the RSNNS package (I am willing to use / evaluate other packages) that's part of R. I would like to get a precision that's at least 66%.

I split the data in a training and test sets, with 4/5 of the data in the training. I then trained models using different network layouts and learning rates, using the same training set each time. I selected the parameters that gave precision >66% and the largest F-measure on the test set.

The parameters I selected gave a precision of 70% on the test set. I then took the data and did a 10-fold cross validation using the same network layout and learning rate. With this k-fold cross validation, I get a precision that is just above 50% (which is similar to the other learning algorithms I used).

My question is, is the 70% precision accurate with the test set? Is my k-fold validation possibly finding local optima, and not giving an accurate precision?

EDIT

Since this seems like an important point I left out, there are 2 classes, positive and negative. It's 18% positive, and 81% negative. There are about 550 cases.

Following Matteo's suggestion in the comments, I ran the network again multiple times. I just used the best selected parameters, because the neural network takes some time to run. I split into training (80%) and test (20%) sets again, except did 10 random splits using sample. Since it's random, some of the data appears more often in the training sets than the test sets. Using this, the precision ranged anywhere from 30% to 70%. When I averaged the 10 runs together, it came out to just above 50% precision.

I am leaning towards saying that is the best precision I can get using this data set, since the earlier machine learning algorithms gave a similar precision (data not shown).

I am attempting to use a neural network, after using other machine learning algorithms. I am using the RSNNS package (I am willing to use / evaluate other packages) that's part of R. I would like to get a precision that's at least 66%.

I split the data in a training and test sets, with 4/5 of the data in the training. I then trained models using different network layouts and learning rates. I selected the parameters that gave precision >66% and the largest F-measure.

The parameters I selected gave a precision of 70%. I then took the data and did a 10-fold cross validation using the same network layout and learning rate. With this k-fold cross validation, I get a precision that is just above 50% (which is similar to the other algorithms I used).

My question is, is the 70% precision accurate with the test set? Is my k-fold validation possibly finding local optima, and not giving an accurate precision?

I am attempting to use a neural network, after using other machine learning algorithms. I am using the RSNNS package (I am willing to use / evaluate other packages) that's part of R. I would like to get a precision that's at least 66%.

I split the data in a training and test sets, with 4/5 of the data in the training. I then trained models using different network layouts and learning rates, using the same training set each time. I selected the parameters that gave precision >66% and the largest F-measure on the test set.

The parameters I selected gave a precision of 70% on the test set. I then took the data and did a 10-fold cross validation using the same network layout and learning rate. With this k-fold cross validation, I get a precision that is just above 50% (which is similar to the other learning algorithms I used).

My question is, is the 70% precision accurate with the test set? Is my k-fold validation possibly finding local optima, and not giving an accurate precision?

EDIT

Since this seems like an important point I left out, there are 2 classes, positive and negative. It's 18% positive, and 81% negative. There are about 550 cases.

Following Matteo's suggestion in the comments, I ran the network again multiple times. I just used the best selected parameters, because the neural network takes some time to run. I split into training (80%) and test (20%) sets again, except did 10 random splits using sample. Since it's random, some of the data appears more often in the training sets than the test sets. Using this, the precision ranged anywhere from 30% to 70%. When I averaged the 10 runs together, it came out to just above 50% precision.

I am leaning towards saying that is the best precision I can get using this data set, since the earlier machine learning algorithms gave a similar precision (data not shown).

Source Link
treed
treed
  • 103
  • 3

test validation versus k-fold cross validation

I am attempting to use a neural network, after using other machine learning algorithms. I am using the RSNNS package (I am willing to use / evaluate other packages) that's part of R. I would like to get a precision that's at least 66%.

I split the data in a training and test sets, with 4/5 of the data in the training. I then trained models using different network layouts and learning rates. I selected the parameters that gave precision >66% and the largest F-measure.

The parameters I selected gave a precision of 70%. I then took the data and did a 10-fold cross validation using the same network layout and learning rate. With this k-fold cross validation, I get a precision that is just above 50% (which is similar to the other algorithms I used).

My question is, is the 70% precision accurate with the test set? Is my k-fold validation possibly finding local optima, and not giving an accurate precision?