# Is backpropagation a frequentist approach?

Frequentist statistics says that there exists some true underlying parameters for a model. While Bayesian statistics says that there are different beliefs about parameters for a model.

In backpropgation, we essentially throw data at a neural network, backpropagate the gradients error, and hope that the network learns the true underlying parameters.

Is training a neural network model with backpropagation, a frequentist approach?

• I disagree that Bayesian statistics does not consider there is a true value for the parameter. – Xi'an Feb 21 '16 at 10:37
• I was taught that there are various "beliefs" for the parameter. These beliefs are not right or wrong. – user46925 Feb 24 '16 at 12:59
• There is no "true" prior, but this does not mean there is no true parameter value behind the observed data. – Xi'an Feb 24 '16 at 13:18

Backpropagation uses a regularizer $\lambda$ to penalize magnitudes of the weights to prevent overfitting. This is a frequentist approach to applying the Occam's Razor principle.
In the Bayesian formulation, this is expressed equivalently as putting a prior distribution on the weights. Your regularizer term $\lambda$ would now be expressed in terms of the prior hyperparameters.
Both approaches work and equivalent in an algorithmic sense. The Bayesian formulation might require a few additional math steps to see how $\lambda$ depends on the prior.
The frequentist approach would use cross-validation to determine $\lambda$ directly. Bayesian's may use cross-validation to fix a prior. Or they may not.