Frameworks for modeling prior knowledge other than Bayesian statistics

It is my understanding that one can easily model prior knowledge about variables or even models with Bayesian statistics. In a certain way, Bayesian stats "forces you" to think about prior knowledge and modeling it explicitly with distributions. It is also my understanding that the only thing that is "fixed" (provided) in Bayesian statistics is "the actual estimator", whereas in frequentist statistics, there are many types of theoretically defined estimators, and much science goes to that.

1. Is my understanding above correct?
2. Are there any other frameworks that help a modeler define prior beliefs explicitly other than Bayesian statistics? Perhaps frameworks that don't require e.g. normalization such as energy-based models? Or is Bayesian statistics truly the only framework where reasoning over prior knowledge is well defined.

There are alternatives, for example, you can use constrained optimization, or regularization. Notice however, that in most cases those approaches can be thought as Bayesian inference in disguise. For example, constraining range of the parameter during optimization, is the same as using flat prior over this range. Using $$L_2$$ regularization is the same as using Gaussian priors.

Moreover, in Bayesian inference you don't need nomalization as well. For both MCMC and optimization, you can work with unnormalized densities. With Approximate Bayesian Computation you can even solve problems where likelihood is not specified as a probability distribution.

Finally, one of the reasons for the popularity of Bayesian approach, is that you end up with a probability distribution for the estimates (posterior), that quantifies uncertainty about the estimates. This is not directly available in other approaches.

• I would be careful about the interpretation of non-Bayesian approaches being Bayesian inference in disguise'; there are many cases where approaches are superficially the same (e.g. the same equations show up), but the inferences drawn are substantially different.
– πr8
Jan 13, 2020 at 17:41
• Perhaps, though I would still argue that this is not really the issue. One of the key misconceptions which drives this perception is the idea that Bayesian inference just means using MAP estimators instead of the MLE, which is not really accurate. One frequently hears the claim that the Lasso is just Bayes with Laplace priors'; this is extremely untrue if one is computing posterior means instead of the MAP.
– πr8
Jan 13, 2020 at 19:40
• @πr8 sure, I totally agree with you. I'm trying to say, that the non-Bayesian approaches to such problems are often quite strongly related to their Bayesian counterparts, & the distion gets blurry.
– Tim
Jan 13, 2020 at 20:47
• @Tim: to hardcore Bayesians, everything that works does so because it's similar to a Bayesian method. I'm not a hardcore Bayesian (i.e., very rarely do I actually use Bayesian methods to analyze data), but I still mostly agree with that idea anyways. Jan 14, 2020 at 17:51
• @πr8 I think you're alluding to the fact that Bayesian inference isn't only about using MAP estimators these days, and that that's been a historical focus because it's easier to compute. If I understand your comment then you're saying it is not true that Lasso can be considered Bayesian when something like Hamiltonian Monte Carlo is used to solve for the full posterior - if you then calculate the mean of the posterior rather than the mode (as in MAP)? Does it "become" Bayesian again, if you look at the mode rather than the mean - even under HMC? I've no idea without a lot more thought! Jan 15, 2020 at 11:40

One way in which prior information can be incorporated into the estimator is through the likelihood (or model, depending on how you look at it). That is to say, when we build a standard parametric model, we are constraining ourselves to say that we are going to allow the model to follow a very specific form, that we know up to the values of the parameters themselves. If we are approximately correct about this form, we should have more efficient estimation than a more general model with more parameters. On the other hand, if our "prior knowledge" is grossly inadquate and this constraint is overly restrictive, we should introduce a lot of bias into our model.

As fairly modern example, Convolutional Neural Networks (CNN) are currently the state of the art for image classification, doing considerably better than vanilla fully connected NN's. The only difference between a CNN and a standard NN is that on the top layers, CNNs only allow for local interactions, where as a fully connected NN doesn't care about how close two pixels are to each other. In other words, the CNN models are a proper subset of the vanilla NNs, where many of the top level parameters are set to 0. This is based on the prior knowledge that nearby pixels are very likely to be related, so by constraining the fully connected model, we get more efficient estimation. Empirically, using this prior information about how we think interactions between pixels should work, we have improved our predictions for image classification.