Frequentist statistics says that there exists some true underlying parameters for a model. While Bayesian statistics says that there are different beliefs about parameters for a model.

In backpropgation, we essentially throw data at a neural network, backpropagate the gradients error, and hope that the network learns the true underlying parameters.

Is training a neural network model with backpropagation, a frequentist approach?

  • 2
    $\begingroup$ I disagree that Bayesian statistics does not consider there is a true value for the parameter. $\endgroup$ – Xi'an Feb 21 '16 at 10:37
  • $\begingroup$ I was taught that there are various "beliefs" for the parameter. These beliefs are not right or wrong. $\endgroup$ – user46925 Feb 24 '16 at 12:59
  • $\begingroup$ There is no "true" prior, but this does not mean there is no true parameter value behind the observed data. $\endgroup$ – Xi'an Feb 24 '16 at 13:18

Backpropagation uses a regularizer $\lambda$ to penalize magnitudes of the weights to prevent overfitting. This is a frequentist approach to applying the Occam's Razor principle.

In the Bayesian formulation, this is expressed equivalently as putting a prior distribution on the weights. Your regularizer term $\lambda$ would now be expressed in terms of the prior hyperparameters.

Both approaches work and equivalent in an algorithmic sense. The Bayesian formulation might require a few additional math steps to see how $\lambda$ depends on the prior.

The frequentist approach would use cross-validation to determine $\lambda$ directly. Bayesian's may use cross-validation to fix a prior. Or they may not.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy