I know that when the prior is conjugate with posterior then one can get an analytical representation for the posterior distribution, but what if these two are not to be conjugate? For example, I would like to use Lasso Bayesian, in which likelihood is the normal distribution and the prior is the Laplace distribution. Then I would like to know, do I still get an approximate sparse solution for the ordinary least square problem?
In most cases we don't have priors that would be conjugate to the likelihood function and result in a closed-form solution. In such a case, we are left with approximate solutions, such as grid approximation for simple problems, and for more realistic ones Laplace approximation, Markov Chain Monte Carlo (MCMC) (see mcmc) sampling, variational inference (see variational-bayes), and others. If you care only about the point estimate of the mode, you can use optimization to obtain the maximum a posterior (MAP) estimate. Those methods have different pros and cons, for example, Laplace approximation and variational inference are fast but less accurate, while MCMC is slow and computationally demanding, but more precise if you draw large enough samples.