2
$\begingroup$

I have a data set (with $n$ points) and there are two models which I think will fit the data well. The first model is a simple power law and the second model is a power law plus a linear term and an intercept;

\begin{eqnarray} y&=&ax^b\\ y&=&ax^b+cx+d. \end{eqnarray}

I would like to use the Bayesian Information Criterion (BIC) to see if using the second model is really worth it. My question is, what exactly is the likelihood $L$ for both of these models? How exactly do I compute $L$ so that I can estimate the BIC?

Improve this question
$\endgroup$
7
  • $\begingroup$ Is your model linear in (all) its parameters ? I mean, is $b$ knowledge-driven or is it a parameter to estimate ? $\endgroup$
    – keepAlive
    Nov 20, 2017 at 21:28
  • $\begingroup$ The likelihood depends on what distribution you assume the errors follow. If you are using a standard software package, often BIC will be calculated for you (perhaps optionally) on the basis of some distributional assumption, e.g., Normally-distributed errors or Poisson regression. $\endgroup$
    – jbowman
    Nov 20, 2017 at 22:09
  • $\begingroup$ @Kanak Both models are nonlinear. I need to estimate $b$ in both cases. Does it matter if my problem is linear or nonlinear? $\endgroup$
    – Fixed Point
    Nov 20, 2017 at 22:14
  • 1
    $\begingroup$ @jbowman I would like to know this in general, for any error distribution. I do have software black-boxes that compute BIC's but my goal is to understand what the calculations really are. How would I do this manually step-by-step? For specificity, for this example my errors are normally distributed with mean zero and variance $\sigma^2$. My data is of the same order of magnitude so normal-error assumption is perfectly valid despite me trying to fit a power-law. $\endgroup$
    – Fixed Point
    Nov 20, 2017 at 22:20
  • 1
    $\begingroup$ You are guessing well. With RSS in hand, one can compute $\text{BIC} = k \ln(n) + n \ln\left( \frac{2\pi \sum_i^nu_i^2}{n}\right) + n$. G. Schwarz. 1978. Estimating the dimension of a model. The Annals of Statistics, pages 461-464. I fully agree with your willing to open black boxes. $\endgroup$
    – keepAlive
    Nov 20, 2017 at 23:09

1 Answer 1

Reset to default
3
$\begingroup$

Actually, a term is missing in your (two) models: the error term. Let's play with the first one:

$y_i = ax_i^b + u_i$ for $i=1,...,n$

The (distribution-specific) likelihood function used in the Bayesian Information Criterion (BIC) is that of $\boldsymbol{u}$ (of the residual vector $\boldsymbol{y} - \widehat{a}\boldsymbol{x}^\widehat{b}=\widehat{\boldsymbol{u}}$ in practice).

This means that you first need to assume a probability density function for $\boldsymbol{u}$, say, $f(\boldsymbol{u},(a,b))$.

Then you can compute -- not to say maximize -- the corresponding log-likelihood function, $\ln(\widehat{L})=\sum_{i=1}^n\ln(f(\widehat{u_i},(\widehat{a},\widehat{b})))$.

Finally, $\text{BIC} = k \ln(n) -2 \ln(\widehat{L})$.

In the (first model) case you describe, $k=2$ and the sensible part is about specifying the probability density function. With your normally distributed residuals, one has

$f(\widehat{u_i},(\widehat{a},\widehat{b})) = (\widehat{\sigma}^22\pi)^{-.5} e^{-.5\widehat{u_i}/\widehat{\sigma}}$

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ The L used in the BIC is that of the residuals, derived from the errors' distribution. This, this was the missing connection. Wikipedia talks about the distribution of the data and I took it to mean the distribution of the (measured) data itself. Now it makes much more sense. +1 and accepted! $\endgroup$
    – Fixed Point
    Nov 21, 2017 at 21:21
  • $\begingroup$ I see the full expression (with the $2\pi$) and the approximation for large $n$ (without the $2\pi$) on wikipedia. Are they both correct? And how large should $n$ be typically before I can use the approximation for large $n$? And these BIC expressions are only valid if the error distributions are from an exponential family? What to do if the error distribution is not from an exponential family? $\endgroup$
    – Fixed Point
    Nov 21, 2017 at 21:23
  • $\begingroup$ @FixedPoint. Relatively to the approximation for large $n$, I am not sure to understand what the authors are explaining. So, the most honest answer I can give you on this, is I do not know. But, for having already re-engineered many statistical packages, I only saw the first expression in use. Your last question is more related to the history of statistics than about statistics themselves, but commonly $30$ is considered as large enough, e.g. see this. $\endgroup$
    – keepAlive
    Nov 21, 2017 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.