New answers tagged

1

The likelihood function is the density of the data considered as a function of the parameter $$L:\,\theta\in\Theta\longmapsto L(\theta;x_1,\ldots,x_n)=\prod_{i=1}^n f_i(x_i;\theta)$$ assuming the random variables $X_i$ in the sample are independent. In the even the $X_i$'s have a finite or countable support, the density is usually defined wrt to the counting ...


0

That would be because cross-entropy measures the Kullback-Leibler divergence of your entire distribution of discrete next token probabilities (to an implicit ground truth distribution). https://en.wikipedia.org/wiki/Cross_entropy Now, normally we are optimizing the MLE described in formula (1) of the paper. So, as to a binary answer to the question in the ...


1

You start at the definition of the (negative) log-likelihood $$\lambda = -\log P = -\log \prod_i p_i = -\sum_i\log p_i$$ where $P$ is the probability of observing the event that you observed (some lamps blew up and others did not). The big event that you observed consists of multiple small events, one for each lamp. Since lamps are independent, big event is ...


1

Your example data is contained in the R package vcd, so I will use it from there. First, the betabinomial distribution has two parameters which we can estimate by maximum likelihood. Let us do that in R: data(Saxony, package="vcd") library(bbmle) # negative loglikelihood function: bb_nloglik0 <- function(x) { function(alpha, beta) { ...


0

TL;DR: to compare different neural network configurations that could have generated the observed training dataset Before talking about neural networks (NNs) let's interpret what the likelihood tells us. For any dataset $\mathcal D=\{x_n\}_n$ of observed variables with an associated probability distribution $p$, i.e., $x_n\sim p({\boldsymbol \theta})$ (where ...


0

We know that we can derive the confidence interval (CI) of the mean for a specified confidence level (CL), based on a data set. Note that the CI actually is the interval such that the real mean is within this interval with the probability equals CL. Back to your first question. To my understanding, your first question is equivalent to ask: based on the data ...


0

It doesn't say that the incomplete-data likelihood is more difficult to optimize! It says that when (in some concrete example) it is more difficult to optimize, and the completed-data likelihood is easy(ier) to maxmize, then you have a good case for using EM. I hope it also gives some example of this!


1

The answer is no; you cannot derive the distribution of the parameter from a knowledge of the data alone. It is important to remember that the prior is the distribution of your beliefs. Imagine that you had beliefs about the location of $\mu$, and they are inconveniently bell-shaped. If they are not flat, then your likelihood will not be proportionate to ...


2

Yes, because if you have two different probabilities $p_1$ and $p_2$ with $p_1 < p_2$, the following are all equivalent: \begin{align} p_1 & < p_2 \\ - p_1 & > - p_2 \\ 1 - p_1 & > 1 - p_2 \\ \frac{1}{1 - p_1} & < \frac{1}{1 - p_2} \end{align} because applying an increasing function does not reverse the inequality (like $f(x) = ...


0

The main reason for using log is to handle very small likelihoods. A 32-bit float can only go down to 2^-126 before it gets rounded to 0. It's not just because optimizers are built to minimize functions, since you can easily minimize -likelihood. If you have a large model computing likelihood of a sequence with hundreds of factors it's easy for likelihood ...


1

Direct answer: The likelihood as described in your second paragraph is defined on the whole of [-inf,inf], and that's the domain on which it should be. Meta-answer: It doesn't matter for any practical inference purpose. Outside of [0.5,1.5], the likelihood will always be multiplied by the zero value of the prior, so you can assign the likelihood any set of ...


Top 50 recent answers are included