Frameworks for modeling prior knowledge other than Bayesian statistics It is my understanding that one can easily model prior knowledge about variables or even models with Bayesian statistics. In a certain way, Bayesian stats "forces you" to think about prior knowledge and modeling it explicitly with distributions. It is also my understanding that the only thing that is "fixed" (provided) in Bayesian statistics is "the actual estimator", whereas in frequentist statistics, there are many types of theoretically defined estimators, and much science goes to that.


*

*Is my understanding above correct?  

*Are there any other frameworks that help a modeler define prior beliefs explicitly other than Bayesian statistics? Perhaps frameworks that don't require e.g. normalization such as energy-based models? Or is Bayesian statistics truly the only framework where reasoning over prior knowledge is well defined.

 A: There are alternatives, for example, you can use constrained optimization, or regularization. Notice however, that in most cases those approaches can be thought as Bayesian inference in disguise. For example, constraining range of the parameter during optimization, is the same as using flat prior over this range. Using $L_2$ regularization is the same as using Gaussian priors.
Moreover, in Bayesian inference you don't need nomalization as well. For both MCMC and optimization, you can work with unnormalized densities. With Approximate Bayesian Computation you can even solve problems where likelihood is not specified as a probability distribution.
Finally, one of the reasons for the popularity of Bayesian approach, is that you end up with a probability distribution for the estimates (posterior), that quantifies uncertainty about the estimates. This is not directly available in other approaches.
A: One way in which prior information can be incorporated into the estimator is through the likelihood (or model, depending on how you look at it). That is to say, when we build a standard parametric model, we are constraining ourselves to say that we are going to allow the model to follow a very specific form, that we know up to the values of the parameters themselves. If we are approximately correct about this form, we should have more efficient estimation than a more general model with more parameters. On the other hand, if our "prior knowledge" is grossly inadquate and this constraint is overly restrictive, we should introduce a lot of bias into our model. 
As fairly modern example, Convolutional Neural Networks (CNN) are currently the state of the art for image classification, doing considerably better than vanilla fully connected NN's. The only difference between a CNN and a standard NN is that on the top layers, CNNs only allow for local interactions, where as a fully connected NN doesn't care about how close two pixels are to each other. In other words, the CNN models are a proper subset of the vanilla NNs, where many of the top level parameters are set to 0. This is based on the prior knowledge that nearby pixels are very likely to be related, so by constraining the fully connected model, we get more efficient estimation. Empirically, using this prior information about how we think interactions between pixels should work, we have improved our predictions for image classification. 
