New answers tagged

1

You are getting infinity if alpha_1 every touches zero, so set the alpha_1 lower boundary to 0.01 or something not zero. I think there is a typo here if(is.infinite(log_like) & log_like>0) log_like <- 1e6, should be 1e-6. set.seed(111) comp <- rbinom(n=1e3,size=1,prob=alpha_actual[1]) Y <- rnorm(1e3, mean=mu_actual[comp+1], sd=sigma_actual[...


0

Turns out: yes, "data-likelihood-weighted regression" is a thing (although it is not called like this). There is actually a field of research that deals particularly with this question and has developed practically feasible solutions. It is called Covariate Shift Adaptation and has been popularized by a series of highly cited papers by Sugiyama et al., ...


1

Showing your optimization code would be helpful as that process should steer clear of unlikely values. For your example with test1, the log likelihood surface looks like the following: You were examining R = 0.02 and k = 0.25 (the point in red) which is far away from the maximum likelihood solution. So if your starting values are around R = 1.5 and k = 1....


1

Firstly, the KL divergence $D_{KL}(p||q)$ is synonymous with the relative entropy, relative entropy of $p$ with respect to $q$. Entropy then becomes the self‐information of a random variable. Mutual information is a special case of a more general quantity called relative entropy, which is a measure of the distance between two probability distributions. ...


8

I will reformulate using my answer at Intuition on the Kullback-Leibler (KL) Divergence. Since I am a statistician, I am more comfortable with likelihoods than with entropies, but I also think that gives more intuition here. Now entropy $ \DeclareMathOperator{\E}{\mathbb{E}} H(X) =-\sum_x p(x) \log p(x) =-\E_p \log p$. So entropy is simply the negative ...


2

formula (1) can be derived as follow : $$\begin{array}[cccc] \ \sum_{i = 1}^{n}(X_i - \mu_0)^2 & = & \sum_{i = 1}^{n}(X_i - \overline{X} + \overline{X} - \mu_0)^2 & (1)\\ & = & \sum_{i = 1}^{n}[(X_i - \overline{X})^2 + (\overline{X} - \mu_0)^2 + 2 (X_i - \overline{X})(\overline{X}-\mu_0)] & (2)\\ & = & \sum_{i = 0}^{n}(X_i ...


1

here is my derivation of formula (1): \begin{equation} \begin{split} \sum_{i=1}^{n}(X_i - \mu_0)^2 & = \sum_{i=1}^{n}(X_i^2-2 X_i \mu_0 + \mu_0^2) \\ & = \sum_{i=1}^{n}X_i^2 - 2 n \mu_0\frac{\sum_{i=1}^{n}X_i}{n} + n\mu_0^2 \\ & = \sum_{i=1}^{n}X_i^2 - 2 n \mu_0 \bar{X} + n \mu_0^2 \\ & = \sum_{i=1}^{n}X_i^2 - n \bar{X}^2 + n (\bar{X} - \...


3

Adding an auxilary $\bar X$ inside the expression: $$\begin{align}\sum^n_{i=1}(X_i-\mu_0)^2&=\sum_{i=1}^n (X_i-\bar{X}+\bar{X}-\mu_0)^2\\&=\sum_{i=1}^n (X_i-\bar{X})^2-2\sum_{i=1}^n(X_i-\bar{X})(\bar{X}-\mu_0)+\sum_{i=1}^n (\bar{X}-\mu_0)^2\\&=\sum_{i=1}^n (X_i-\bar{X})^2 + 2(\bar X - \mu_0)\overbrace{\sum_{i=1}^n (X_i-\bar{X})}^0 + n(\bar X-\...


2

For a random sample of iid normal RV's the MLE of the $E[X_1]$ is the sample mean. Hence, the MLE of $\hat{\phi}$ is simply the sample mean, $\bar{X}$. Then $$\gamma = P(X>0) = 1-\Phi(0-\phi),$$ where $\Phi(\cdot)$ is the standard normal cdf. By invariance of MLE, it follows that the MLE of $\gamma$ is $$\hat{\gamma} = 1-\Phi(-\hat{\phi}) = 1-\Phi(-\...


0

Yes. If $z^*$ maximizes $p(x|z^*)$, then the optimal distribution for $p^*(z)$ is a one-hot distribution that assigns all of its weight to $z^*$, i.e., $p^*(z^*)=1$ and $p^*(z)=0$ for all $z \ne z^*$.


0

For this particular case, yes, it works. If $p(x|0)<p(x|1)$, then $z^*=1$, and $$p_{\theta^*}(x) = p(x|0) (1-\theta^*) + p(x|1) \theta^* = p(x|0) + (p(x|1)-p(x|0))\theta^*,$$ which is maximized at $\theta^*=1$. Similarly, if $p(x|0)>p(x|1)$, then $z^*=0$ and $\theta^*=0$.


2

You are going down the correct path --- when you are looking for the UMVUE in a parametric problem, the simplest method in most cases is to use the Lehmann–Scheffé theorem, which says that if you can form an unbiased estimator from a complete sufficient statistic, then that estimator is the unique UMVUE. Now, from your stipulated distribution, you get the ...


2

The key point is that $\sum\limits_1^n k_j$ is a sufficient statistic for estimating the parameter $\mu$ of a Poisson distribution. More precisely, the conditional probability distribution of seeing the data $k_1,k_2,\ldots,k_n$, given that $\sum\limits_1^n k_j=s$ for any non-negative integer $s$, does not depend on the parameter $\mu$. So nothing more ...


1

So you have a Laplace distribution with mean $0$. This is not difficult. The likelihood is $$\prod \frac{1}{2\sigma} e^{-|x_i|/\sigma} = \frac{1}{2^n\sigma^n} e^{-\sum |x_i| /\sigma} $$ and the log-likelihood is $$-n \log_e(2) - n \log_e(\sigma) - \frac{\sum |x_i|}{\sigma} $$ and the derivative of the log-likelihood with respect to $\sigma$ is $$-\frac{...


3

Here's a way to compute the log likelihood, avoiding numerical under/overflow by working with log probabilities and using the log-sum-exp trick. Let $p_i(\lambda)$ be the product of the probabilities over $j$, given $i$ and $\lambda$: $$p_i(\lambda) = \prod_j f(i,j,\lambda)$$ Obviously, the log can be computed as: $$\log p_i(\lambda) = \sum_j \log f(i,j,\...


Top 50 recent answers are included