Can Bayesian Models be used to Compensate for Small Datasets?

Question

I have the following question: Can Bayesian Models be used to Compensate for Small Datasets?

Suppose we have a linear regression problem (e.g. predict the age of giraffes based on their height and weight) where we were not able to collect sufficient data, but we have have access to historical information from similar problems that we believe might be useful for defining the priors within Bayes Law. I tried to manually write the estimation equations for a Bayesian Linear Regression problem:

In this type of problem, suppose we were only able to collect measurements on a very small number of giraffes - could the Bayesian Priors in theory (if they indeed happen to accurately reflect the real data) be used to compensate for the small datasets and serve to "push" the estimates of the linear regression model towards more realistic values?

In a Frequentist setting, we are limited by the data we collect: if our data has poor quality (e.g. missing data, inaccurate data, small data), the model is almost guaranteed to suffer. But as far as I understand, perhaps Bayesian Models might be able to overcome this problem by strategically exploiting historical information from previous studies (e.g. meta-analysis) and turning them into priors?

Thanks

Answer 1

Take as an example the simple beta-binomial model, where we observed $k$ successes in $n$ trials and want to learn the probability of success $p$. We start with a beta prior for $p$.

$$\begin{align} p &\sim \mathsf{Beta}(\alpha, \beta) \\ k &\sim \mathsf{Binomial}(n, p) \end{align}$$

in such a case, the posterior distribution would be

$$ p|k,n \sim \mathsf{Beta}(\alpha + k, \, \beta + n - k) $$

with the expected value

$$ E[p] = \frac{\alpha + k}{\alpha + \beta + n} $$

Notice that in such a case the $\alpha$ and $\beta$ hyperparameters can be thought as pseudocounts, so before seeing the data we assume to have observed $\alpha$ successes and $\beta$ failures. We are augmenting the data with the pseudocounts.

While it may be less obvious, the same thing happens when you use other Bayesian models. By assuming priors you start with pre-calculated parameters that are updated by applying Bayes theorem. The more data you gather, the less impactful the priors would be. If you have good prior knowledge, it may help with overcoming some drawbacks of the data. On another hand, as noticed in comments by @Dave, if you start with priors that are wrong, they would also bias your estimates.