8
$\begingroup$

So, for OLS there are 3 assumptions regarding the DGP, which are (from Stock & Watson):

  • Independence of error terms (+ Homoskedasticity?)
  • IID of variables
  • Large outliers are unlikely, meaning non-zero finite 4th moments

I'm currently working with Bayesian Hierarchical Linear Models, and I always thought that the "philosophical" part in Bayesianism is justifying the prior. Are the set of assumptions needed for frequentist linear regression as strict as in bayesian linear regression and/or are there even different sets of assumptions?

$\endgroup$
1

2 Answers 2

7
$\begingroup$

Let me use the linear regression example, that you mentioned. The simple linear regression model is

$$ y_i = \alpha + \beta x_i + \varepsilon_i $$

with noise being independent, normally distributed random variables $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$. This is equivalent of stating the model in terms of normal likelihood function

$$ y_i \sim \mathcal{N}(\alpha + \beta x_i, \;\sigma^2) $$

The assumptions that we make follow from the probabilistic model that we defined:

  • we assumed that the model is linear,
  • we assumed i.i.d. variables,
  • variance $\sigma^2$ is the same for every $i$-th observation, so the homoscedasticity,
  • we assumed that the likelihood (or noise, in first formulation) follows normal distribution, so we do not expect to see heavy tails etc.

Plus some more "technical" things like no multicollinearity, that follow from the choice of method for estimating the parameters (ordinary least squares).

(Notice that those assumptions are needed for things like confidence intervals, and testing, not for the least squares linear regression. For details check What is a complete list of the usual assumptions for linear regression? )

The only thing that changes with Bayesian linear regression, is that instead of using optimization to find point estimates for the parameters, we treat them as random variables, assign priors for them, and use Bayes theorem to derive the posterior distribution. So Bayesian model would inherit all the assumptions we made for frequentist model, since those are the assumptions about the likelihood function. Basically, the assumptions that we make, are that the likelihood function that we've chosen is a reasonable representation of the data.

As about priors, we do not make assumptions about priors, since priors are our a priori assumptions that we made about the parameters.

$\endgroup$
14
  • 1
    $\begingroup$ Do you have any published references for this? Especially "The only thing that changes..." and "...Bayesian model would inherit all the assumptions...". $\endgroup$
    – Mankka
    Commented Nov 9, 2019 at 20:33
  • 1
    $\begingroup$ @Mankka reference for what exactly? In maximum likelihood you maximize likelihood, in Bayesian approach you care about likelihood * prior, so the only element that differs is assuming that parameters are random variables and using priors. As about inheriting assumptions, the assumptions are about likelihood function (besides the "technical" ones). As a reference could serve any Bayesian statistics handbook. $\endgroup$
    – Tim
    Commented Nov 9, 2019 at 21:26
  • $\begingroup$ Thanks. I've read a few books about bayesian data analysis but none of them really divided into the assumptions. Do you have any recommendations? $\endgroup$
    – RazorLazor
    Commented Nov 9, 2019 at 23:16
  • $\begingroup$ @RazorLazor people don't mention assumptions of Bayesian models because they are stated directly when defining likelihood and priors. On another hand, if you're fitting a line using least squares it is less obvious what exactly are the assumptions you made, so it is worth discussing them directly. $\endgroup$
    – Tim
    Commented Nov 9, 2019 at 23:46
  • $\begingroup$ @Tim yeah it makes sense and i understand it now but I'd like to cite it that's why i asked. $\endgroup$
    – RazorLazor
    Commented Nov 10, 2019 at 19:04
6
$\begingroup$

Assumptions in bayesian statistics are generally stronger than those, because you need, in every model, to specify the full distribution of your data and parameters.

In many cases, gaussian distribution is used, because of its relation to expected value and arithmetic mean, without really believing in the assumption of normality, and it has been shown that the results are quite robust to departures from normality, in case the same conditions as above are respected.

One other example of a distribution used in bayesian statistics even if data is not really believed to follow it, is asymmetric Laplace, for quantile regression. Bayesian models are very varied, I don't know which are you talking about, but most probably it's gaussian ones. In that case, if you respect the same assumptions as for frequentist models, you should be ok (homoskedasticity is one of those, unless heteroskedasticity is explicitly addressed).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.