# Using the BIC to select a model

I have a data set (with $n$ points) and there are two models which I think will fit the data well. The first model is a simple power law and the second model is a power law plus a linear term and an intercept;

\begin{eqnarray} y&=&ax^b\\ y&=&ax^b+cx+d. \end{eqnarray}

I would like to use the Bayesian Information Criterion (BIC) to see if using the second model is really worth it. My question is, what exactly is the likelihood $L$ for both of these models? How exactly do I compute $L$ so that I can estimate the BIC?

• Is your model linear in (all) its parameters ? I mean, is $b$ knowledge-driven or is it a parameter to estimate ? – keepAlive Nov 20 '17 at 21:28
• The likelihood depends on what distribution you assume the errors follow. If you are using a standard software package, often BIC will be calculated for you (perhaps optionally) on the basis of some distributional assumption, e.g., Normally-distributed errors or Poisson regression. – jbowman Nov 20 '17 at 22:09
• @Kanak Both models are nonlinear. I need to estimate $b$ in both cases. Does it matter if my problem is linear or nonlinear? – Fixed Point Nov 20 '17 at 22:14
• @jbowman I would like to know this in general, for any error distribution. I do have software black-boxes that compute BIC's but my goal is to understand what the calculations really are. How would I do this manually step-by-step? For specificity, for this example my errors are normally distributed with mean zero and variance $\sigma^2$. My data is of the same order of magnitude so normal-error assumption is perfectly valid despite me trying to fit a power-law. – Fixed Point Nov 20 '17 at 22:20
• You are guessing well. With RSS in hand, one can compute $\text{BIC} = k \ln(n) + n \ln\left( \frac{2\pi \sum_i^nu_i^2}{n}\right) + n$. G. Schwarz. 1978. Estimating the dimension of a model. The Annals of Statistics, pages 461-464. I fully agree with your willing to open black boxes. – keepAlive Nov 20 '17 at 23:09

Actually, a term is missing in your (two) models: the error term. Let's play with the first one:

$y_i = ax_i^b + u_i$ for $i=1,...,n$

The (distribution-specific) likelihood function used in the Bayesian Information Criterion (BIC) is that of $\boldsymbol{u}$ (of the residual vector $\boldsymbol{y} - \widehat{a}\boldsymbol{x}^\widehat{b}=\widehat{\boldsymbol{u}}$ in practice).

This means that you first need to assume a probability density function for $\boldsymbol{u}$, say, $f(\boldsymbol{u},(a,b))$.

Then you can compute -- not to say maximize -- the corresponding log-likelihood function, $\ln(\widehat{L})=\sum_{i=1}^n\ln(f(\widehat{u_i},(\widehat{a},\widehat{b})))$.

Finally, $\text{BIC} = k \ln(n) -2 \ln(\widehat{L})$.

In the (first model) case you describe, $k=2$ and the sensible part is about specifying the probability density function. With your normally distributed residuals, one has

$f(\widehat{u_i},(\widehat{a},\widehat{b})) = (\widehat{\sigma}^22\pi)^{-.5} e^{-.5\widehat{u_i}/\widehat{\sigma}}$

• The L used in the BIC is that of the residuals, derived from the errors' distribution. This, this was the missing connection. Wikipedia talks about the distribution of the data and I took it to mean the distribution of the (measured) data itself. Now it makes much more sense. +1 and accepted! – Fixed Point Nov 21 '17 at 21:21
• I see the full expression (with the $2\pi$) and the approximation for large $n$ (without the $2\pi$) on wikipedia. Are they both correct? And how large should $n$ be typically before I can use the approximation for large $n$? And these BIC expressions are only valid if the error distributions are from an exponential family? What to do if the error distribution is not from an exponential family? – Fixed Point Nov 21 '17 at 21:23
• @FixedPoint. Relatively to the approximation for large $n$, I am not sure to understand what the authors are explaining. So, the most honest answer I can give you on this, is I do not know. But, for having already re-engineered many statistical packages, I only saw the first expression in use. Your last question is more related to the history of statistics than about statistics themselves, but commonly $30$ is considered as large enough, e.g. see this. – keepAlive Nov 21 '17 at 21:57