# MLE for the logistic distribution

The log likelihood is given by

$$\log L(k,\mu) = \sum_{i=1}^n \log k - k(x_i-\mu) - 2\log(1+e^{-k(x_i-\mu)})$$

If $$-k(x_i-\mu)$$ is large, enough $$\log(1+e^{-k(x_i-\mu)}\approx -k(x_i - \mu)$$.

This means that $$\log L(k,\mu) \approx \sum_{i=1}^n \log k + k(x_i-\mu)$$.

which would be unbounded if $$k\to\infty,\mu\to-\infty$$.

My question is how can I make this end up working? I would prefer a gradient based way to do parameter estimation.

• The way you write the likelihood, $k$ is the reciprocal of the scale parameter, hence $$k \to \infty \implies {\rm var} \to 0 \implies {\rm pdf} \to 0 \quad {\rm and}\,\, {\rm CDF} \to 1.$$ If the data have very small variance/variability, what realistically do you expect to get out of them? Commented Jul 22 at 18:36
• should i use the scale parameter then? Commented Jul 22 at 18:39

#### Try doing your expansion more formally (and cancel low-order terms)

Your expansion is rather vague and does not appear to consider possible cancellation of lower-order terms. If you do the expansion formally then you ought to obtain a sensible solution where a low-order expansion of the log-likelihood still has sensible properties for maximisation.

To see this, let's expand formally using Taylor series representation of the tricky term. Taking $$\mu$$ as the mean parameter and $$k$$ as the rate (inverse-scale) parameter, you have the log-likelihood:

\begin{align} \ell_\mathbf{x}(\mu) &= \sum_{i=1}^n \log \text{Logistic}(x_i|\mu,k) \\[6pt] &= \sum_{i=1}^n \Bigg[ \log(k) - k(x_i-\mu) - 2 \cdot \text{log1pexp}(-k(x_i-\mu)) \Bigg] \\[6pt] &= n \log(k) - k \sum_{i=1}^n (x_i-\mu) -2 \sum_{i=1}^n \text{log1pexp}(-k(x_i-\mu)). \\[6pt] \end{align}

Taking the Taylor expansion of the $$\text{log1pexp}$$ term around $$\mu$$ gives the series representation:$$^\dagger$$

\begin{align} \text{log1pexp}(-k(x_i-\mu)) &= \log(2) - \frac{k (x_i-\mu)}{2} + \frac{k^2 (x_i-\mu)^2}{8} - \frac{k^4 (x_i-\mu)^4}{192} + \cdots \end{align}

so we have:

\begin{align} \ell_\mathbf{x}(\mu) &= n \log(k) - k \sum_{i=1}^n (x_i-\mu) -2 \sum_{i=1}^n \bigg[ \log(2) - \frac{k (x_i-\mu)}{2} + \frac{k^2 (x_i-\mu)^2}{8} + \cdots \bigg] \\[6pt] &= n \log(k) - k \sum_{i=1}^n (x_i-\mu) - 2n \log(2) + k \sum_{i=1}^n (x_i-\mu) - \sum_{i=1}^n \frac{k^2 (x_i-\mu)^2}{4} + \cdots \\[6pt] &= n [\log(k) - 2 \log(2)] - \sum_{i=1}^n \frac{k^2 (x_i-\mu)^2}{4} + \cdots \\[6pt] \end{align}

As you can see, there is a cancellation of the first-order term in the expansion so the next biggest term is the second-order term. Ignoring the higher order terms, and taking a second-order polynomial approximation to the log-likelihood, we get the approximating MLEs:

$$\hat{\mu} = \bar{x}_n \quad \quad \quad \quad \quad \hat{k} = \sqrt{\frac{n-1}{2n}} \cdot \frac{1}{s_n},$$

where $$\bar{x}_n$$ and $$s_n$$ are the sample mean and sample standard deviation of the data respectively. These are sensible approximations for the MLE and the situation does not appear to be problematic or unusual. These would also make sensible starting points for the parameters when solving the full MLE using numerical methods (e.g., using Newton-Raphson iteration).

$$^\dagger$$ We have the first four derivatives:

\begin{align} D_1(x) \equiv \frac{d}{dx} \text{log1pexp}(-k(x-\mu)) &= - \frac{k e^{-k(x-\mu)}}{1+e^{-k(x-\mu)}}, \\[6pt] D_2(x) \equiv \frac{d^2}{dx^2} \text{log1pexp}(-k(x-\mu)) &= \frac{k^2 e^{-k(x-\mu)}}{(1+e^{-k(x-\mu)})^2}, \\[6pt] D_3(x) \equiv \frac{d^3}{dx^3} \text{log1pexp}(-k(x-\mu)) &= - \frac{k^3 e^{-k(x-\mu)}(1-e^{-k(x-\mu)})}{(1+e^{-k(x-\mu)})^3}, \\[6pt] D_4(x) \equiv \frac{d^4}{dx^4} \text{log1pexp}(-k(x-\mu)) &= \frac{k^4 e^{-k(x-\mu)}(1-4e^{-k(x-\mu)}+e^{-2k(x-\mu)})}{(1+e^{-k(x-\mu)})^4}, \\[6pt] \end{align}

and taking $$x=\mu$$ gives the simplification $$e^{-k(x-\mu)} = 1$$ which then yields the values:

\begin{align} D_1(\mu) &= - \frac{k}{2}, \\[6pt] D_2(\mu) &= \frac{k^2}{4}, \\[12pt] D_3(\mu) &= 0, \\[10pt] D_4(\mu) &= - \frac{k^4}{8}. \\[6pt] \end{align}

This gives the Taylor expansion:

\begin{align} \text{log1pexp}(-k(x-\mu)) &= \text{log1pexp}(0) + \sum_{r=1}^\infty \frac{D_r(\mu)}{r!} (x-\mu)^r \\[6pt] &= \log(2) - \frac{k (x-\mu)}{2} + \frac{k^2 (x-\mu)^2}{8} - \frac{k^4 (x-\mu)^4}{192} + \cdots \\[6pt] \end{align}

I think there's something wrong with your calculation. When I compute the loglikelihood for a particular $$x_i$$ and for very high precision and very negative location I get very negative loglikelihood

> dlogis(0,location=-10,scale=0.0001,log=TRUE)
[1] -99990.79
> dlogis(0,location=-100,scale=0.0000001,log=TRUE)
[1] -1e+09


so it seems to converge to $$-\infty$$ as I'd expect rather than $$+\infty$$.

There is (as with $$N(\mu,\sigma^2)$$ and many other location-scale models) an unbounded-above log likelihood when the sample size is equal to 1, with $$\mu=x_1$$, and $$k\to \infty$$. This is fixed by having more than one point.

• yeah there's something wrong... the approximation log(1+e^-x) ~ -x is true when -x is large, not when x is large. Commented Jul 22 at 20:04

If $$-k(x_i-\mu)$$ is large, enough $$\log(1+e^{-k(x_i-\mu)})\approx -k(x_i - \mu)$$.

This means that $$\log L(k,\mu) \approx \sum_{i=1}^n \log k + k(x_i-\mu)$$.

This approximation is correct but it requires $$-k(x_i - \mu) \gg 0$$, or stated differently $$k(x_i - \mu) \ll 0$$.

So you showed that $$\log L(k,\mu) \to -\infty$$ as $$\mu \to \infty$$.

You applied this approximation for $$\mu \to \infty$$ to $$\mu \to -\infty$$. That leads to the false conclusion.

For the other direction, $$\mu \to -\infty$$, you get $$\log(1+e^{-k(x_i-\mu)})\approx \log(1) = 0$$ and $$\log L(k,\mu) \approx \sum_{i=1}^n \log k - k(x_i-\mu)$$, which has the opposite sign from your expression.