# Tag Info

1

You are getting infinity if alpha_1 every touches zero, so set the alpha_1 lower boundary to 0.01 or something not zero. I think there is a typo here if(is.infinite(log_like) & log_like>0) log_like <- 1e6, should be 1e-6. set.seed(111) comp <- rbinom(n=1e3,size=1,prob=alpha_actual[1]) Y <- rnorm(1e3, mean=mu_actual[comp+1], sd=sigma_actual[...

0

Turns out: yes, "data-likelihood-weighted regression" is a thing (although it is not called like this). There is actually a field of research that deals particularly with this question and has developed practically feasible solutions. It is called Covariate Shift Adaptation and has been popularized by a series of highly cited papers by Sugiyama et al., ...

1

Showing your optimization code would be helpful as that process should steer clear of unlikely values. For your example with test1, the log likelihood surface looks like the following: You were examining R = 0.02 and k = 0.25 (the point in red) which is far away from the maximum likelihood solution. So if your starting values are around R = 1.5 and k = 1....

1

Firstly, the KL divergence $D_{KL}(p||q)$ is synonymous with the relative entropy, relative entropy of $p$ with respect to $q$. Entropy then becomes the self‐information of a random variable. Mutual information is a special case of a more general quantity called relative entropy, which is a measure of the distance between two probability distributions. ...

8

I will reformulate using my answer at Intuition on the Kullback-Leibler (KL) Divergence. Since I am a statistician, I am more comfortable with likelihoods than with entropies, but I also think that gives more intuition here. Now entropy $\DeclareMathOperator{\E}{\mathbb{E}} H(X) =-\sum_x p(x) \log p(x) =-\E_p \log p$. So entropy is simply the negative ...

2

formula (1) can be derived as follow : $$\begin{array}[cccc] \ \sum_{i = 1}^{n}(X_i - \mu_0)^2 & = & \sum_{i = 1}^{n}(X_i - \overline{X} + \overline{X} - \mu_0)^2 & (1)\\ & = & \sum_{i = 1}^{n}[(X_i - \overline{X})^2 + (\overline{X} - \mu_0)^2 + 2 (X_i - \overline{X})(\overline{X}-\mu_0)] & (2)\\ & = & \sum_{i = 0}^{n}(X_i ... 1 here is my derivation of formula (1): \begin{split} \sum_{i=1}^{n}(X_i - \mu_0)^2 & = \sum_{i=1}^{n}(X_i^2-2 X_i \mu_0 + \mu_0^2) \\ & = \sum_{i=1}^{n}X_i^2 - 2 n \mu_0\frac{\sum_{i=1}^{n}X_i}{n} + n\mu_0^2 \\ & = \sum_{i=1}^{n}X_i^2 - 2 n \mu_0 \bar{X} + n \mu_0^2 \\ & = \sum_{i=1}^{n}X_i^2 - n \bar{X}^2 + n (\bar{X} - \... 3 Adding an auxilary \bar X inside the expression:$$\begin{align}\sum^n_{i=1}(X_i-\mu_0)^2&=\sum_{i=1}^n (X_i-\bar{X}+\bar{X}-\mu_0)^2\\&=\sum_{i=1}^n (X_i-\bar{X})^2-2\sum_{i=1}^n(X_i-\bar{X})(\bar{X}-\mu_0)+\sum_{i=1}^n (\bar{X}-\mu_0)^2\\&=\sum_{i=1}^n (X_i-\bar{X})^2 + 2(\bar X - \mu_0)\overbrace{\sum_{i=1}^n (X_i-\bar{X})}^0 + n(\bar X-\...

2

For a random sample of iid normal RV's the MLE of the $E[X_1]$ is the sample mean. Hence, the MLE of $\hat{\phi}$ is simply the sample mean, $\bar{X}$. Then $$\gamma = P(X>0) = 1-\Phi(0-\phi),$$ where $\Phi(\cdot)$ is the standard normal cdf. By invariance of MLE, it follows that the MLE of $\gamma$ is $$\hat{\gamma} = 1-\Phi(-\hat{\phi}) = 1-\Phi(-\... 0 Yes. If z^* maximizes p(x|z^*), then the optimal distribution for p^*(z) is a one-hot distribution that assigns all of its weight to z^*, i.e., p^*(z^*)=1 and p^*(z)=0 for all z \ne z^*. 0 For this particular case, yes, it works. If p(x|0)<p(x|1), then z^*=1, and$$p_{\theta^*}(x) = p(x|0) (1-\theta^*) + p(x|1) \theta^* = p(x|0) + (p(x|1)-p(x|0))\theta^*,$$which is maximized at \theta^*=1. Similarly, if p(x|0)>p(x|1), then z^*=0 and \theta^*=0. 2 You are going down the correct path --- when you are looking for the UMVUE in a parametric problem, the simplest method in most cases is to use the Lehmann–Scheffé theorem, which says that if you can form an unbiased estimator from a complete sufficient statistic, then that estimator is the unique UMVUE. Now, from your stipulated distribution, you get the ... 2 The key point is that \sum\limits_1^n k_j is a sufficient statistic for estimating the parameter \mu of a Poisson distribution. More precisely, the conditional probability distribution of seeing the data k_1,k_2,\ldots,k_n, given that \sum\limits_1^n k_j=s for any non-negative integer s, does not depend on the parameter \mu. So nothing more ... 1 So you have a Laplace distribution with mean 0. This is not difficult. The likelihood is$$\prod \frac{1}{2\sigma} e^{-|x_i|/\sigma} = \frac{1}{2^n\sigma^n} e^{-\sum |x_i| /\sigma} $$and the log-likelihood is$$-n \log_e(2) - n \log_e(\sigma) - \frac{\sum |x_i|}{\sigma} $$and the derivative of the log-likelihood with respect to \sigma is$$-\frac{...

3

Here's a way to compute the log likelihood, avoiding numerical under/overflow by working with log probabilities and using the log-sum-exp trick. Let $p_i(\lambda)$ be the product of the probabilities over $j$, given $i$ and $\lambda$: $$p_i(\lambda) = \prod_j f(i,j,\lambda)$$ Obviously, the log can be computed as: \log p_i(\lambda) = \sum_j \log f(i,j,\...

Top 50 recent answers are included