Difference between cross-validation and back propagation

Question

sorry if this is a dumb question. I want to know what the difference is? Mainly what how they're objectives differ from each other?

I know that in cross validation you divide the training set into equal parts and use one as a test set. Back propagation means that the output is sent in reverse of sorts to recalculate the weights of the neural network.

But wouldn't they achieve the same thing? Is it that cross-validation deals with the actual output against future data, while back propagation is for weight optimization?

Answer 1

Is it that cross-validation deals with the actual output against future data, while back propagation is for weight optimization?

Yes, cross validation is meant to assess if your model performs well on data that was not used to build it.

Backpropagation in neural networks is an iterative method by which you train/optimize weights to improve classification or regression accuracy.

Answer 2

Cross validation implies that the data is compared with a subset of the data (or, less frequently, an external data source). In this regard, it is a "replication" within the data, but with more room for uncertainty. The original idea is to test for consistency - does the inference from the data hold also for smaller parts? In machine learning, the result is improved accuracy.

Back propagation is originally a phenomenon observed in real neurons, due to the mechanics of the axon membrane. However, it has a very specific meaning in computational neuroscience:

The procedure repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector. As a result of the weight adjustments, internal ‘hidden’ units which are not part of the input or output come to represent important features of the task domain, and the regularities in the task are captured by the interactions of these units.

Rumelhart, Hinton & Williams (1986). [Abstract] Learning representations by back-propagating errors, Nature, 323, 533-536. doi: 10.1038/323533a0 http://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html

Answer 3

This is not a dumb question, and a good question to ask.

Neural network learns from the data that is provided while the training is happening. A portion of data is not used in the training process and kept for the later stage called the validation/test stage. The idea behind is that, normally in the neural network, a complete learning process would include a feed forward and eventually a backword propagation, where the neural network validates if learning is done right and if not it adjusts the weights accordingly. After this complete cycle, the weights are ready for the prediction and the final output from the model. In validation phase, the actual trained model is tested and validated on the trained model to see if the model really learned well. Since the validation data is unseen by the neural network model and hence the hypothesis should output a minimized error. In this scenario we can call the model is near to perfect model and we have verified that with the unseen data.

Now a point to be noted is, there is no adjustment of the weights in this later process and no backward prop. So we can also call it as a half cycle of neural network learning process. And hence the term Backward Propagation vs Cross Validation. Hope it helps, let me know if any question.