37

apparently the Bayes factor somehow uses likelihoods that represent the likelihood of each model integrated over it's entire parameter space (i.e. not just at the MLE). How is this integration actually achieved typically? Does one really just try to calculate the likelihood at each of thousands (millions?) of random samples from the parameter space, or are ...


25

I second Nick Sabbe's harsh comment, and my short answer is, It is not. I mean, it only is in the normal linear model. For absolutely any other sort of circumstances, the exact distribution is not a $\chi^2$. In many situations, you can hope that Wilks' theorem conditions are satisfied, and then asymptotically the log-likelihood ratio test statistics ...


23

As mentioned by @Nick this is a consequence of Wilks' theorem. But note that the test statistic is asymptotically $\chi^2$-distributed, not $\chi^2$-distributed. I am very impressed by this theorem because it holds in a very wide context. Consider a statistical model with likelihood $l(\theta \mid y)$ where $y$ is the vector observations of $n$ independent ...


19

R.V. Foutz and R.C. Srivastava has examined the issue in detail. Their 1977 paper "The performance of the likelihood ratio test when the model is incorrect" contains a statement of the distributional result in case of misspecification alongside a very brief sketch of the proof, while their 1978 paper "The asymptotic distribution of the likelihood ratio when ...


18

The LR (likelihood ratio) test actually is testing the hypothesis that a specified subset of the parameters equal some pre-specified values. In the case of model selection, generally (but not always) that means some of the parameters equal zero. If the models are nested, the parameters in the larger model that are not in the smaller model are the ones ...


18

I'm not sure there's really a canonical answer to this, but I'll give it a shot. What is the recommended way to select the best fitting model in this context? When using log-likelihood ratio tests what is the recommended procedure? Generating models upwards (from null model to most complex model) or downwards (from most complex model to null model)? ...


18

AIC and likelihood ratio test (LRT) have different purposes. AIC tells you whether it pays to have a richer model when your goal is approximating the underlying data generating process the best you can in terms of Kullback-Leibler distance. LRT tells you whether at a chosen confidence level you can reject the hypothesis that some restrictions on the ...


16

As mentioned in the previous answer, the difference comes down to a difference in scaling, i.e., different estimators for the standard deviation of the errors. Sources for the difference are (1) scaling by $n-k$ (the unbiased OLS estimator) vs. scaling by $n$ (the biased ML estimator), and (2) using the estimator under the null hypothesis or alternative. ...


15

The required regularity conditions are listed in most intermediate textbooks and are not different than those of the mle. The following ones concern the one parameter case yet their extension to the multiparameter one is straightforward. Condition 1: The pdfs are distinct, i.e. $\theta \neq \theta ^{\prime} \Rightarrow f(x_i;\theta)\neq f(x_i;\theta ^{\...


15

It's important to note that although the likelihood ratio test and the Wald test are used by researchers to accomplish the same empirical goal(s), they are testing different hypotheses. The likelihood ratio test evaluates whether the data were likely to have come from a more complex model, vs. a more simple model. Put another way, does the addition of a ...


15

If one takes the logarithm of this product, $${\mathfrak{r}}=\log \prod_{i=1}^n \frac{f(x_i)}{g(x_i)} = \sum_{i=1}^n \log\frac{f(x_i)}{g(x_i)}$$and turns it into an average $$\bar{\mathfrak{r}}_n=\frac{1}{n}\sum_{i=1}^n \log\frac{f(x_i)}{g(x_i)}$$the law of large numbers applies, hence one gets the almost sure convergence $$\bar{\mathfrak{r}}_n\stackrel{\...


13

Your third item is the one I have seen the most often used as rigorous definition. The others are interesting too (+1). In particular the first is appealing, with the difficulty that the sample size not being (yet) defined, it is harder to define the "from" set. To me, the fundamental intuition of the likelihood is that it is a function of the model + its ...


13

A very short answer: the REML is a ML, so the test based on REML is correct anyway. As the estimation of the variance parameters with REML is better, it is natural to use it. Why is REML a ML? Consider e.g. a model $$Y = X\beta + Zu + e \def\R{\mathbb{R}}$$ with $X\in\R^{n\times p}$, $Z\in\R^{n\times q}$, and $\beta \in \R^p$ is the vector of the fixed ...


12

The Poisson and negative binomial (NB) model are nested: Poisson is a special case with theta = infinity. So a likelihood ratio test comparing the two models is testing the null hypothesis that "theta = infinity" against the alternative that "theta < infinity". Here the two models have the following log-likelihoods R> logLik(m3) 'log Lik.' -1328.642 (...


12

Update May 2017: As it turns out, a lof of what I have written here is kind of wrongish. Some updates are made throughout the post. I agree a lot with what has been said by Ben Bolker already (thanks for the shout-out to afex::mixed()) but let me add a few more general and specific thoughts on this issue. Focus on fixed versus random effects and how to ...


11

The statistical term deviance is thrown around a bit too much. Most of the time, programs return the deviance $$D(y) = -2\log{\{p(y\textrm{ }|\hat{\theta})\}},$$ where $\hat{\theta}$ is your estimated parameter(s) from model fitting and $y$ is some potentially observed/observable occurrence of the random quantity in question. The more common deviance that ...


11

In understanding the difference between likelihood ratios and Bayes factors, it is useful to consider one key feature of Bayes factors in more detail: How do Bayes factors manage to automatically account for the complexity of the underlying models? One perspective on this question is to consider methods for deterministic approximate inference. Variational ...


11

For logistic regression you use the asymptotic distribution of the log of likelihood ratio test statistic for variable selection (testing hypotheses or model selection). In the case of linear regression, due to the assumed normality for the error distribution, there is no need to use asymptotics, and the likelihood ratio test static trivially reduces to a ...


11

I will use the same notation I used here: Mathematics behind classification and regression trees Gini Gain and Information Gain ($IG$) are both impurity based splitting criteria. The only difference is in the impurity function $I$: $\textit{Gini}: \mathit{Gini}(E) = 1 - \sum_{j=1}^{c}p_j^2$ $\textit{Entropy}: H(E) = -\sum_{j=1}^{c}p_j\log p_j$ They ...


11

I recently wrote an entry in a linkedin blog stating Neyman Pearson lemma in plain words and providing an example. I found the example eye opening in the sense of providing a clear intuition on the lemma. As often in probability, it is based on a discrete probability mass function so it is easy to undertand than when working with pdf's. Also, take into ...


11

They are not the same, but in you case they could be used for the same purpose. Optimal Bayes classifier is $$ \DeclareMathOperator*{\argmax}{arg\,max} \argmax_{c \in C} p(c|X) $$ i.e., among all hypotheses, take the $c$ that maximizes the posterior probability. You use Bayes theorem $$ \underbrace{p(c|X)}_{\text{posterior}} \propto \underbrace{p(X|c)}_{...


10

The main problem is that if you're going to use the ratio as your response variable, you should be using the weights argument. You must have ignored a warning about "non-integer #successes in a binomial glm" ... Dilution <- c(1/128, 1/64, 1/32, 1/16, 1/8, 1/4, 1/2, 1, 2, 4) NoofPlates <- rep(x=5, times=10) NoPositive <- c(0, 0, 2, 2, 3, 4, 5, 5, 5,...


10

You have a few issues here. First, understanding what each test is doing, and second interpreting the p-values. First, each test has different underlying assumptions. The likelihood ratio test statistic is formed by taking the log of the ratio of the likelihood under the null model, divided by the alternative model. The test statistic is approximately ...


9

The derivation of AIC as an estimator of Kullback-Leibler information loss makes no assumptions of models being nested.


9

The paper Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 307-333. has the full theoretical treatment and test procedures. It distinguishes between three situations, "Strictly Non-nested Models", "Overlapping Models", "Nested Models", and also examines cases of misspecification. It is therefore no-...


8

I think I would call it something different. Likelihood is the probability density for the observed x given the value of the parameter $θ$ expressed as a function of $θ$ for the given $x$. I don't share the view about the proportionality constant. I think that only comes into play because maximizing any monotonic function of the likelihood gives the same ...


8

The paper by Koopman (1984) Confidence intervals for the ratio of two binomial proportions gives two methods for calculating the confidence interval. I am gonna explain the first one here as the confidence intervals can be calculated analytically (the second method uses an iterative procedure to find the confidence intervals numerically). First, consider the ...


8

No, the likelihood ratio is not the only way to construct hypothesis tests, but it often is optimal. In one flavour of the frequentist paradigm you can construct a hypothesis test from any arbitrary test statistic that can generate a p value ie a probability of observing the data, given the null hypothesis. An alternative hypothesis does not need to be ...


8

Disclaimer: As in the comments, these are not ways to ensure best prediction, but rather the musings of an epidemiologist on model building for survival models trying to elucidate the relationship between an outcome O and an exposure E with a number of covariates: The goal of these is not actually to make the best predictive model, or the strongest ...


Only top voted, non community-wiki answers of a minimum length are eligible