Alternatives to Bayesian statistics when distributions are unknown In Bayesian statistics, with my variable is Gaussian distributed and I have a conjugate prior, I can solve the posterior analytically. I can still use MCMC in the case when things are non-Gaussian so long as I nominate a distribution. But what if I don't know what the appropriate distribution is? I am working with financial data which are known to be non-Gaussian (heavier tails and skewed relative to a Normal distribution). To the best of my knowledge, the exact distribution of financial data is still up for debate in academia. What would be an alternative if I don't want to make a strong assumption on how the data is distributed?
 A: Although the other answers are ok, I think they might be overkill for yours or for many other problems.
Without changing the paradigm, if you know the distribution has heavier tails than Gaussian, you can fit t-distribution with either fixed heaviness of tails or with the heaviness of tails that are estimated from the data possibly with a prior information.
If a distribution is skewed and has heavier tails, you can fit a SHASH distribution which has four parameters parametrizing location, scale, skewness, and kurtosis of the distribution with the normal distribution as its special case.
A: If the distribution of the data is unknown, the Bayesian way of handling this uncertainty is to put a prior on it. There exists a huge literature on Bayesian non-parametrics, including the Fundamentals of Nonparametric Bayesian inference by Ghosal and van der Vaart. The default priors in such settings are distributions on distributions, like Dirichlet processes. Check e.g. the webpage Tutorials on Bayesian Nonparametrics, maintained by Peter Orbanz. Here is a list of seminal papers given to our students.
Concerning MCMC, there exist MCMC algorithms that handle Bayesian nonparametrics as well. See e.g. this book by Dey et al. Check also the dirichlet-process tag on this forum.
A milder solution is to use Bayesian model averaging, that is, to list all (!) the plausible families that could fit the data, choose a reference prior on each, and use the posterior mixture for quantities of interest [common to all families].
A: From what you’re saying is that you want something Bayesian, but you can’t define the likelihood. For such cases there is approximate Bayesian computation (see abc), where in place of likelihood you use some summary statistics.
As a side note, for using proper Bayesian analysis you don’t need to know the “exact” distribution. We nearly never do. You need to use some distribution that relatively well approximates the distribution of the data. This is how is it done in most of the statistics. We don’t use Gaussian, Poisson, etc distributions because they are the exact distributions of the observed data, but they are good enough approximations for the purpose. Same you do with loss function, you don’t use squared error because it has some deep meaning for your data, but because it works well enough.
A: I suggest as alternative bootstrap based methods, which can be parametric and not, and works with very few assumptions and very few data (only 5-7 samples suffice); it works on non gaussian distributions and for many measures (eg., mean, etc). One can "rebuild" a distribution by bootstraping the mean, if it makes sense in the given context. The Davison1 is good for introducting the bootstrap in general.
Many papers were published on the volatility smile analysed throught bootstrap based methods [0].


*https://scholar.google.com/scholar?q=bootstrap+volatility+smile

*Davison, A.C. and Hinkley, D.V., 1997. Bootstrap methods and their application (No. 1). Cambridge university press.

*The Non-parametric Bootstrap as a Bayesian Model
