# Random-walk prior with ridge-like regularizarion?

I am working with a model that contains a large number of coefficients, arranged in an ordered vector $$\beta_1, \dots, \, \beta_N$$. I have some prior knowledge that could be used to improve the estimation of the $$\beta$$, in particular:

1. There's autocorrelation in the vector of $$\beta$$, such that the difference between contiguous values, e.g. $$\beta_i$$ and $$\beta_{i+1}$$, is more likely to be 'small', rather than 'large';
2. I expect most $$\beta$$ coefficients to be unimportant; specifically I expect only a single, relatively small (compared to $$N$$), cluster of contiguous coefficient to be different from zero and useful for predicting my dependent variable.

I am working in a Bayesian setting (I use Stan). My idea was to include both these information in the model using a combination of a random-walk prior (to enforce smoothness and correlation between contiguous $$\beta$$) with a regularizing prior, to give higher prior probability to small values of $$\beta$$.

More in detail, I have a random walk prior such that $$\beta_1 \sim \mathcal{N}(0,1), \quad \beta_i \sim \mathcal{N}(\beta_{i-1}, \tau)$$ where $$\tau$$ is a smoothness hyperparameter.

At the same time I want to penalize large values of $$\beta$$. In ridge regression this would be achieved by adding a penalty $$-\lambda \sum_{i=1}^N \beta_i^2$$ to the log-likelihood. In a Bayesian setting this should be equivalent to setting a zero-centred Gaussian prior, e.g. $$\beta_i \sim \mathcal{N}\left(0, \frac{1}{2\lambda}\right)$$

Putting everything together, this would give the following modified/regularized random walk prior: $$\beta_1 \sim \mathcal{N}\left(0, \: \frac{1}{1+2\lambda}\right), \quad \beta_i \sim \mathcal{N}\left(\frac{\frac{1}{2\lambda}}{\tau + \frac{1}{2\lambda}}\beta_{i-1}, \: \frac{1}{\frac{1}{\tau} + 2\lambda} \right)$$

My question is: does this approach makes sense or are there shortcomings/limitations that I have missed? Is there any reference I can cite that used similar approach? Or alternatively, is there a better way to do this?

Thanks!

• are you familiar with the fused lasso? Would those ideas be helpful?
– jld
Commented Nov 27, 2019 at 17:22
• I think you’re on the right track but would use something else for the actual prior. See here for example: arxiv.org/pdf/1706.06908.pdf Commented Nov 27, 2019 at 17:41
• Thanks, I did not know about the fused lasso (and Bayesian fused lasso mentioned in the link), I will look into it. Commented Nov 27, 2019 at 17:52
• i don't understand why you don't set $\beta_1 \sim N(0,\tau_1)$ and $\beta_i \sim N(\beta_{i-1},\tau)$. your initial setup is already regularising. remember that you can pull the $\beta_{i-1}$ out of the mean. so $\beta_i= \beta_1 + \sum_{1<j<=i} N(0,\tau)$. ie you are regularising beta 1 - and the consecutive diffs. Commented Nov 30, 2019 at 18:51