What's the proper way of finding the best prediction model function for the data?

Question

What's the best way of predicting variable $z$ given variables ${a_1, a_2, ..., a_n}$?

I'm using a Bayesian Model to predict variable $z$. I have parameters $a_i$ that I have inferred a prior distribution:

$$a_1 = a_2 = a_3 \sim Normal(0,100)$$ $$a_4 = a_5 \sim Uniform(100,200)$$ $$a_6 = a_7 \sim Beta(1,1)$$ $$a_6 = a_8 \sim Gamma(1,1)$$

But I'm having 2 problems:

1. I can't figure out a way of finding the likelihood function. The output $z$ consists of data the are very concentrated around zero (some negative values too) and have long tail for positive values. Should I just say it looks like an exponential distribution and assume that distribution as my likelihood function?
2. How should I relate all those n parameters to predict $z$? Is it like try-and-error? Like, I guess that it could be linear relationship so I say $z = a_1 + b_1*a_2 + ... + b_n*a_n$ and predict $z$, and if that is not good I try another model?

Answer 1

A few things that might help:

1. The distribution of $z$ by itself isn't likely to be important, but the distribution of your data given values of $b_i$ will be. That may sound like a pedantic difference but it's a big one.

   For example, say you've chosen to use a gaussian likelihood and you have 10 observations of $z$ for which $b_i = 0, \forall i\in[1, n]$ (i.e. all the predictors are 0) and then 10 other observations for which $b_i = 1, \forall i\in[1, n]$. Then, as long as the distribution of $z$ around the predicted mean in the first 10 cases (which would be 0) is roughly gaussian and the and distribution of $z$ around the second 10 with a predicted mean equal to the sum of the coefficients is also roughly gaussian, you'd have a good model (even though the distributions of the $z$ values across all 20 observations is something weird -- a mixture of two gaussians). Even in this simple case just looking at the $z$ values themselves wouldn't tell you anything useful. You'd be better off picking one of several likelihoods based on some theoretical reasoning and then comparing the fit of them all to choose the best one.

2. With regards to both choosing a likelihood and a form for the model, you'll want to know how to do one of two things (or possibly both):

• Model Selection - There are many ways to compare models to one another for the sake of choosing the "best" one, but using an information criteria like WAIC is a good choice. Bayes factors or other information criteria like AIC are other possibilities, but WAIC generally has better properties. Regardless though, these things are always just approximations of how much better one model is likely to be at making predictions so you could also just test that directly through cross-validation too.

• Model Averaging - Rather than choose a single model, you can also provide some inference using all many of them. This is possible for the sake of making predictions as well as determining covariate effects, though it takes a good bit of computational legwork.

  The first option is certainly more common (you could characterize it as "trial-and-error" in a sense) so I'd start there if you don't think a simple linear model with nothing more than main effects is good enough (there's no way to know a priori how complicated the model should be).

Not knowing anything more about your data or problem (but at least assuming the outcome is continuous), I'll take a somewhat wild guess and say that either a gaussian or gamma likelihood will give you about as reasonable of a model as you're likely to get with more pedestrian methods.

Failing that you could also punt on picking a likelihood and try Approximate Bayesian Computing instead. There are "likelihood-free" ways to draw inference like this, though they seem to be used much less frequently.

Answer 2

Regarding 1: I just wrote an answer in your other question that may or may not be useful. In short, yes you typically want to make some distributional assumption about your data via the likelihood.

Regarding 2: it doesn't make sense to have a bunch of parameters floating around without knowing where to put them. The parameters are inputs to either either the likelihood or other distributions that make up the structure of your model.

As an example, if your likelihood is $\text{binomial}(n, p)$ for some known $n$, then the only parameter of your model is $p$.

You could go further backwards from there; say $p \sim \text{beta}(\alpha, \beta)$, so that you'd have two other parameters $\alpha$ and $\beta$ involved in your model. But the point is that you don't usually have a bunch of parameters floating around with no knowledge of where to put them: if that's the case then it means you haven't really specified a model for your data.