6
$\begingroup$

According to this wikipedia article, score is the derivative of the log-likelihood function. However, I don't understand what if we have two parameters? For example, the logarithm of pdf has the following form: $$ln f(x|a,b) = a\cdot g(x) + b \cdot h(x) + \phi(a,b)$$ I think that this is the sum of partial derivatives i.e.

$$s = g(x) + h(x) + \phi(a,b)_{a}' + \phi(a,b)_{b}'$$

but I'm not sure.

$\endgroup$

1 Answer 1

12
$\begingroup$

The score for a multiple parameter problem (a vector parameter) is itself a vector. We need to take partial derivatives of the log likelihood with respect to each model parameter.

Let's consider an example. Find the score vector for $X_1, X_2, \ldots, X_n \sim N(\mu, \sigma^2)$ where the $X_i$ are iid $N(\mu, \sigma^2)$ samples. Let $\mathbf{x} = (x_1, x_2, \ldots, x_n)$ be a random sample of $(X_1, X_2, \ldots, X_n)$.

In this case, it can be shown that the log likelihood is

$$\ell(\mu, \sigma | \mathbf{x}) = -\frac{n}{2}\log (2 \pi) - n\log\sigma - \frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2$$

So differentiation w.r.t. $\mu$ gives

$$ \frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2} \sum_{i=1}^n (x_i - \mu)$$ differentiation w.r.t. $\sigma$ gives $$ \frac{\partial \ell}{\partial \sigma} = -\frac{n}{\sigma} + \frac{1}{\sigma^3}\sum_{i=1}^n (x_i - \mu)^2.$$

The score is then the vector $S(\mu, \sigma) = \left( \frac{\partial \ell}{\partial \mu}, \frac{\partial \ell}{\partial \sigma} \right)^T$.

Concisely, we can write $S(\mathbf{\theta}) = \nabla_{\mathbf{\theta}} \ell(\mathbf{\theta}|\mathbf{x})$ for any vector parameter $\mathbf{\theta}$ where $\ell(\theta |\mathbf{x})$ is the log likelihood function.

$\endgroup$
1
  • 1
    $\begingroup$ In Langevin dynamics, we see the score function $\nabla_{\mathbf{x}}\ \log p(\mathbf{x})$. But is that the same thing as this score? The $\nabla_{\mathbf{x}}$ seems to be w.r.t the data and not a parameter vector, e.g one is w.r.t the $x$ in the support and the other is w.r.t the mean and std deviation, etc. $\endgroup$
    – flinty
    Commented Dec 2, 2023 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.