I currently sample LASSO and Horseshoe regression in STAN. Hence I was wondering how to properly define the hyperpriors in the bayesian regression models. I.e. Park and Casella use a gamma prior with hyperparameter r, d.
Some sources state that it is reasonable to choose low values (i.e. a relative flat prior for as suggested by Park & Casella 2008). That does make sense to me, as high values lead to rather narrow marginal densities of beta.
Hence, wouldn't it make sense to directly use the marginal likelihood of beta?
lambda2 ~ Gamma(r, d)
tau ~ Exp(lambda2 / 2)
beta ~ Normal(0, tau)
So we can sample first lambda, then tau and finally beta?
r = 1
d = 1.1
lambda <- matrix(rgamma(1000, r, d), ncol = 1)
tau <- apply(lambda, 2, function(x) rexp(1000, x / 2))
beta <- apply(tau, 2, function(x) rnorm(1000, 0, x))
plot(density(beta))
Then check if we like the regularization or not? length(which(beta < -6)) / length(beta)
I know that if I just choose lambda to be flat enough and then sample very long it doesn't matter too much which r & d I choose. Yet I am looking for a more scientific answer.
Thanks for your thoughts.
Note that you have to fix the seed so that the comparison is not dependent on it.