9
$\begingroup$

Say we have a Bayesian polynomial regression like the following.

$$y_i \sim N(\mu_i, \sigma^2)$$ $$\mu_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i^3 $$

where $x_i$ is some mean centred predictor variable.

I'm interested in what kind of prior to specify that is reasonably uninformative. While I generally find a uniform prior to be adequate for the linear coefficient $\beta_1$, I imagine uniform priors on the quadratic $\beta_2$ and cubic $\beta_3$ coefficients might be problematic because small increases can have very large effect on predictions. Thus, I was thinking that perhaps some sort of highly skewed distribution that makes values closer to zero more likely might be more appropriate.

  • Is there a standard choice of noninformative priors for quadratic and cubic parameters in polynomial regression?
  • Or alternatively, is there a good default strategy for choosing priors for such parameters (e.g., perhaps based on features of x, y or their relationship) ?
Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ A small movement in either direction (positive or negative) "can have very large effect on predictions". Thus, a skewed prior doesn't seem like the right way to go. It sounds like you are interested in a light-tailed option. $\endgroup$
    – gung - Reinstate Monica
    Commented Mar 17, 2015 at 3:34
  • $\begingroup$ @gung Perhaps I'm overthinking it and uniform or diffuse normal priors are okay. It might be that my current modelling issues which motivated the question are caused by other issues (e.g., poor starting values). $\endgroup$
    – Jeromy Anglim
    Commented Mar 17, 2015 at 3:45
  • 2
    $\begingroup$ As stated there is no reason to process this problem differently from another regression. I would thus go for Zellner's $g$-prior which is both noninformative and handles most naturally the various scales of the regressors. $\endgroup$
    – Xi'an
    Commented Mar 17, 2015 at 8:30

2 Answers 2

Reset to default
4
$\begingroup$

To be honest, in the absence of a clear theory, I'd be inclined to frame the model in terms of orthogonal polynomials, and then put priors on the higher order terms that indicate that I expect them to be small -- which would tend to pull the posterior toward zero, rather than be uninformative.

If I wanted an uninformative prior, I'd still be tempted to do it with orthogonal polynomials.

Edit: Elvis' question about why orthogonal polynomials is important --

One big advantage of orthogonal polynomials is that (because of the orthogonality) lower order coefficients are not affected if the higher order coefficients are shrunk or even set to 0. It makes it more sensible to try to do things like order selection with averaging of parameter estimates across models. (It's less important if interest is only on the function as a whole, but sometimes the values of particular coefficients, or functions of them, can be important.)

(There may also be some advantages in accuracy and computational effort; in particular if you're doing all possible degrees of polynomial, there can be an advantage in effort.)

The polynomials are orthogonal in the following sense - $\sum_i p_j(x_i)\,p_k(x_i)=0$ (when $j\neq k$).

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ Thanks. That's helpful. So it sounds like the approach you are recommending is broadly consistent with my intuition (i.e., that large values are implausible given a basic knowledge of the data and therefore a prior should somewhat incorporate that knowledge). And perhaps instead of "non-informative", I'm really trying to ask, what is a good default approach. $\endgroup$
    – Jeromy Anglim
    Commented Mar 17, 2015 at 5:45
  • $\begingroup$ What is the point of using orthogonal polynomials? And orthogonal... with respect to which inner product? $\endgroup$
    – Elvis
    Commented Mar 17, 2015 at 7:02
  • 1
    $\begingroup$ @Elvis - I've made an addition to my answer. $\endgroup$
    – Glen_b
    Commented Mar 17, 2015 at 8:41
  • 1
    $\begingroup$ @Elvis - sorry, right now I can't say I know of anything that makes this connection, though I'm quite sure it's been done more than once. [We all have things we don't know, my own knowledge is deficient in so many areas (but those are usually the ones I'm not answering questions in so you don't notice quite how much I don't know) -- that's one of the reasons I like coming here, to learn off others.] $\endgroup$
    – Glen_b
    Commented Mar 17, 2015 at 22:25
  • 1
    $\begingroup$ @Elvis correct, it's equivalent to that. $\endgroup$
    – Glen_b
    Commented Mar 20, 2015 at 9:12
1
$\begingroup$

Have you tried to use Jeffrey's prior? This is a standard method for generating non-informative priors (though I admit I have no experience with complicated likelihood functions you have mentioned)

http://en.wikipedia.org/wiki/Jeffreys_prior

The basic idea is that a non-informative prior will contain no information on transformation and relates to the Fisher information which I don't think should not be too complicated to find (finding posteriors is another issue).

Apoligies if you have already tried this!

A slight addendum to my previous answer

It just occurred to me that the problem is a standard linear regression

Consider a matrix $Z$ whose $i^{th}$ row is given by $[1, x_{i}, x_{i}^{2}....]$. Your model is essentially

$ \mu = Z\beta + \epsilon$

where $\epsilon\sim$ $N(0,\sigma^{2})$

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.