2
$\begingroup$

I am working with a model that contains a large number of coefficients, arranged in an ordered vector $\beta_1, \dots, \, \beta_N $. I have some prior knowledge that could be used to improve the estimation of the $\beta$, in particular:

  1. There's autocorrelation in the vector of $\beta$, such that the difference between contiguous values, e.g. $\beta_i$ and $\beta_{i+1}$, is more likely to be 'small', rather than 'large';
  2. I expect most $\beta$ coefficients to be unimportant; specifically I expect only a single, relatively small (compared to $N$), cluster of contiguous coefficient to be different from zero and useful for predicting my dependent variable.

I am working in a Bayesian setting (I use Stan). My idea was to include both these information in the model using a combination of a random-walk prior (to enforce smoothness and correlation between contiguous $\beta$) with a regularizing prior, to give higher prior probability to small values of $\beta$.

More in detail, I have a random walk prior such that $$ \beta_1 \sim \mathcal{N}(0,1), \quad \beta_i \sim \mathcal{N}(\beta_{i-1}, \tau) $$ where $\tau$ is a smoothness hyperparameter.

At the same time I want to penalize large values of $\beta$. In ridge regression this would be achieved by adding a penalty $-\lambda \sum_{i=1}^N \beta_i^2$ to the log-likelihood. In a Bayesian setting this should be equivalent to setting a zero-centred Gaussian prior, e.g. $$ \beta_i \sim \mathcal{N}\left(0, \frac{1}{2\lambda}\right) $$

Putting everything together, this would give the following modified/regularized random walk prior: $$ \beta_1 \sim \mathcal{N}\left(0, \: \frac{1}{1+2\lambda}\right), \quad \beta_i \sim \mathcal{N}\left(\frac{\frac{1}{2\lambda}}{\tau + \frac{1}{2\lambda}}\beta_{i-1}, \: \frac{1}{\frac{1}{\tau} + 2\lambda} \right) $$

My question is: does this approach makes sense or are there shortcomings/limitations that I have missed? Is there any reference I can cite that used similar approach? Or alternatively, is there a better way to do this?

Thanks!

$\endgroup$
4
  • 1
    $\begingroup$ are you familiar with the fused lasso? Would those ideas be helpful? $\endgroup$
    – jld
    Nov 27 '19 at 17:22
  • 1
    $\begingroup$ I think you’re on the right track but would use something else for the actual prior. See here for example: arxiv.org/pdf/1706.06908.pdf $\endgroup$
    – hejseb
    Nov 27 '19 at 17:41
  • $\begingroup$ Thanks, I did not know about the fused lasso (and Bayesian fused lasso mentioned in the link), I will look into it. $\endgroup$
    – matteo
    Nov 27 '19 at 17:52
  • $\begingroup$ i don't understand why you don't set $\beta_1 \sim N(0,\tau_1)$ and $\beta_i \sim N(\beta_{i-1},\tau)$. your initial setup is already regularising. remember that you can pull the $\beta_{i-1}$ out of the mean. so $\beta_i= \beta_1 + \sum_{1<j<=i} N(0,\tau)$. ie you are regularising beta 1 - and the consecutive diffs. $\endgroup$
    – seanv507
    Nov 30 '19 at 18:51
1
$\begingroup$

I would look into the horseshoe-type markov random fields (MRFs) described in https://projecteuclid.org/euclid.ba/1487905413. They generalize Gaussian MRFs to include horseshoe-type priors which induce the type of shrinkage of the coefficients you want, while maintaining the autoregressive properties from a random walk. They also have a Stan implementation!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.