1
$\begingroup$

I have been running ANN (Neural network) on my data set, until last week that I figured out I will get more robust model using Cross validation. So that's why I have started using ANN with the aid of cross validation.

For example using a 10-fold cross validation, all the dataset will be divided into 10 sunsets and each time one of the subsets is being used as test set while the rest is being used as training set. so 10 models in being built based on the 10-fold cross validation, and at the end we have all them combined in one model (?). Regarding all these process I was expecting a smaller R-sq (r-sq obtained from predicted values vs. actual values) using "ANN with cross validation" Vs. "ANN without Cross validation". but the strange thing is the Cross validation ANN R-sq are bigger than the simple ANN (without cross validation ANN) ones. I was thinking that maybe sth is going wrong. I do not understand how, could you please help me with it?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Cross validation is mainly used for evaluation purposes (for instance no clearly defined train/test split, a desire to calculate statistical significance, etc.) When making a final model, it would make more sense to train on the entire data set and not average the weights - see:

Averaging weights learned during backpropogation

Averaging weights can sometimes be useful though - like in averaged perceptron, but just not in the case of NNs. Note that in training NNs what is often done is to hold out some data and after each epoch test on this held out set. When performance on the set decreases then you are beginning to overfit and should stop training

Improve this answer
$\endgroup$
6
  • $\begingroup$ I am using Cross validation and Neural Networks in SAS Enterprise miner, I do not build the models manually, so in the end node sas gives me a columns of predicted values regarding the combined model, $\endgroup$
    – user36107
    Dec 12, 2013 at 20:50
  • $\begingroup$ the good thing about cross validation (k=10) is that it uses each subset once as test set, so for k=10 at the end of the story we have 10 models regarding 10 test subsets , and all the models get together and make one combined model. if I am not mistaken you are saying that it is possible that using cross validation lead to obtaining bigger R-sq? $\endgroup$
    – user36107
    Dec 12, 2013 at 21:12
  • $\begingroup$ Bigger R sq than what? What is the alternative? Are you testing than on your training data? $\endgroup$
    – user671931
    Dec 12, 2013 at 22:21
  • $\begingroup$ I am running neural networks with and without cross validation, my expectation is that with the same neural network properties I get lower R-sq in the ANN with cross validation than ANN wothout cross validation. what do you think? $\endgroup$
    – user36107
    Dec 12, 2013 at 23:03
  • 1
    $\begingroup$ I am confused about how you are evaluating the ANN without cross validation. You must be evaluating on something...For cross validation you are averaging R-sq from each round - without cross validation - how are you obtaining your R sq value? $\endgroup$
    – user671931
    Dec 12, 2013 at 23:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.