1
$\begingroup$

I have the following question: Can Bayesian Models be used to Compensate for Small Datasets?

Suppose we have a linear regression problem (e.g. predict the age of giraffes based on their height and weight) where we were not able to collect sufficient data, but we have have access to historical information from similar problems that we believe might be useful for defining the priors within Bayes Law. I tried to manually write the estimation equations for a Bayesian Linear Regression problem:

enter image description here

In this type of problem, suppose we were only able to collect measurements on a very small number of giraffes - could the Bayesian Priors in theory (if they indeed happen to accurately reflect the real data) be used to compensate for the small datasets and serve to "push" the estimates of the linear regression model towards more realistic values?

In a Frequentist setting, we are limited by the data we collect: if our data has poor quality (e.g. missing data, inaccurate data, small data), the model is almost guaranteed to suffer. But as far as I understand, perhaps Bayesian Models might be able to overcome this problem by strategically exploiting historical information from previous studies (e.g. meta-analysis) and turning them into priors?

Thanks

$\endgroup$
3
  • 3
    $\begingroup$ The Brandon Rohrer YouTube video on Bayes’ theorem gets into how that can be done. Keep in mind that, just like an awesome prior can get out of trouble when we have little data, a bad prior is going cause problems. $\endgroup$
    – Dave
    Nov 5, 2021 at 2:42
  • $\begingroup$ Thank you for your reply! I will watch this video! $\endgroup$
    – stats_noob
    Nov 5, 2021 at 2:57
  • $\begingroup$ great, now everytime a coworker suggests bayesian I will joke with "are you compensating for something?" $\endgroup$
    – Adrian
    Jun 6 at 8:12

1 Answer 1

1
$\begingroup$

Take as an example the simple beta-binomial model, where we observed $k$ successes in $n$ trials and want to learn the probability of success $p$. We start with a beta prior for $p$.

$$\begin{align} p &\sim \mathsf{Beta}(\alpha, \beta) \\ k &\sim \mathsf{Binomial}(n, p) \end{align}$$

in such a case, the posterior distribution would be

$$ p|k,n \sim \mathsf{Beta}(\alpha + k, \, \beta + n - k) $$

with the expected value

$$ E[p] = \frac{\alpha + k}{\alpha + \beta + n} $$

Notice that in such a case the $\alpha$ and $\beta$ hyperparameters can be thought as pseudocounts, so before seeing the data we assume to have observed $\alpha$ successes and $\beta$ failures. We are augmenting the data with the pseudocounts.

While it may be less obvious, the same thing happens when you use other Bayesian models. By assuming priors you start with pre-calculated parameters that are updated by applying Bayes theorem. The more data you gather, the less impactful the priors would be. If you have good prior knowledge, it may help with overcoming some drawbacks of the data. On another hand, as noticed in comments by @Dave, if you start with priors that are wrong, they would also bias your estimates.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.