4
$\begingroup$

Well normally proving identifiability follows by showing that $p_{\theta}(x)=p_{\theta'}(x)$ implies $\theta=\theta'$. Usually this proceeds by showing that a function dependent on $\theta$, such as log-likelihood (or conditional log-likelihood in the case of Logistic Regression) leading to equality implies the equality $\theta=\theta'$. In that sense it is like showing the likelihood function is one-to-one when applied to a single data point. To be more precise for a dataset of size one $\mathcal{D}=\{X\}$ from Gaussian distribution, to show identifiability we write: $$nll_{\mu, \sigma}(\mathcal{D})=c + \log(\sigma) + \frac{1}{2{\sigma}^2}(X-\mu)^2=c + \log(\sigma') + \frac{1}{2{\sigma'}^2}(X-\mu')^2$$. Now taking the derivative twice with respect to $X$ we get $$\frac{1}{\sigma^2}=\frac{1}{{\sigma'}^2}$$ which then implies $\sigma=\sigma'$. Conclusion about $\mu=\mu'$ also follows with similar algebraic manipulations.

My question is what if we imply identifiablity for the likelihood of a large set of points rather than one point only, and in doing so, we could incorporate limiting laws such as LLN. For example in the case of identifiability of Gaussian distribution we can do as follows. Again assume $X_i\sim \mathcal{N}(\mu, \sigma)$.

Now if we have $$ nll_\mu(\mathcal{D})=c + N\log(\sigma) + \frac{1}{2{\sigma}^2}\sum_{i=1}(X_i-\mu)^2=c +N\log(\sigma')+ \frac{1}{2{\sigma'}^2}\sum_{i=1}(X_i-\mu')^2 =nll_{\mu'}(\mathcal{D})$$ dividing both sides with $N$ and taking the limit of $N\to\infty$ which implies using SLLN: $$\log(\sigma)+\frac{1}{2}\overset{P}{=} \log(\sigma')+\frac{1}{2}$$ which implies $\sigma=\sigma'$.

So rather than using a single point, I am using a large sample of points to show identifiability. Is this allowed? If so where can I find this formally written in a textbook? If not, why not? Isn't the point to show that when large enough sample is gathered we can identify the true parameter anyway?

$\endgroup$
3
  • 1
    $\begingroup$ Because the point of LLN (weak or strong) is to show convergence to a particular constant. If you want some other result about convergence ("to a large set of points"), then please be more specific. You may have something very useful in mind, but it's not clear (to me) what. $\endgroup$
    – BruceET
    Aug 6 '20 at 22:31
  • $\begingroup$ I understand that, and I am using here the knowledge of this squared summation converging to a particular constant to conclude identifiability of normal distributions. I am trying to understand if I am allowed to do this, but I'll add more detais. $\endgroup$
    – Cupitor
    Aug 7 '20 at 1:23
  • $\begingroup$ @BruceET to emphasize again what I care about is identifiability the whole attempt is to use LLN as a tool to prove identifiability by relying on inifinitely large sample size. $\endgroup$
    – Cupitor
    Aug 7 '20 at 1:40
3
$\begingroup$

Ultimately, the problem here is that $\sigma \stackrel{P}{=} \sigma' $ does not imply that $\sigma = \sigma'$, so no, you cannot. A couple of notes though:

Well normally proving identifiability follows by showing that $p_{\theta}(x)=p_{\theta'}(x)$ implies $\theta=\theta'$.

Since the definition of identifiability is that the function $\theta \mapsto p_\theta$ is injective (one-to-one), every proof must ultimately show that $p_{\theta}(x)=p_{\theta'}(x)$ implies $\theta=\theta'$, since that is the definition of an injective function. There might be different ways to arrive at this conclusion, but it is a necessity for the proof.

[this] is like showing the likelihood function is one-to-one when applied to a single data point.

This is true, but nothing about the function $\theta \mapsto p_\theta$ has anything to do with data points, it's simply the case that the two look the same analytically when there is one data point. The function $\theta \mapsto p_\theta$ is determined only by how we decided to represent a certain set of distributions by their parameters. It's all a priori and not based on any kind of data collection whatsoever.

Addendum: Okay, let's go through it formally. Let $\mathscr{L}$ denote the negative log-likelihood, which we need to view as a function not only of the parameters $\mu$ and $\sigma$, but also as a random variable, i.e. as a function of $\omega \in \Omega$ where $\Omega$ is the sample space: $$ \mathscr{L}_{\mu , \sigma} (\omega) = c + N \log(\sigma) + \frac{1}{2\sigma^2} \sum_{i=1}^N (X_i (\omega) - \mu)^2 $$ Start with the assumption that $p_{\mu, \sigma} = p_{\mu', \sigma'}$. Since we have two distributions, we should have two samples, say $X_i$ and $Y_i$, and we know that $E(X_i) = \mu$, $V(X_i) = \sigma^2$, $E(Y_i) = \mu'$, and $V(Y_i) = \sigma'^2$. We can cancel the constants $c$ and write \begin{align} N \log(\sigma) + \frac{1}{2\sigma^2} \sum_{i=1}^N (X_i (\omega) - \mu)^2 & = N \log(\sigma') + \frac{1}{2\sigma'^2} \sum_{i=1}^N (Y_i (\omega) - \mu')^2 \\ N \log(\sigma) + \frac{N}{2\sigma^2} \left(\frac{1}{N}\right) \sum_{i=1}^N (X_i (\omega) - \mu)^2 & = N \log(\sigma') + \frac{N}{2\sigma'^2} \left(\frac{1}{N}\right) \sum_{i=1}^N (Y_i (\omega) - \mu')^2 \end{align} and now we want to apply the weak law of large numbers (WLLN) (since you are using the notation for convergence in probability, not almost sure convergence like you would get from using the strong law of large numbers) to both sides which tells us that \begin{align} \left(\frac{1}{N}\right) \sum_{i=1}^N (X_i (\omega) - \mu)^2 & \stackrel{P}{\rightarrow} E[(X_i (\omega) - \mu)^2] = \sigma^2 \\ \left(\frac{1}{N}\right) \sum_{i=1}^N (Y_i (\omega) - \mu')^2 & \stackrel{P}{\rightarrow} E[(Y_i (\omega) - \mu')^2] = \sigma'^2. \end{align} Let's remember what convergence in probability means though: $$ \left(\frac{1}{N}\right) \sum_{i=1}^N (X_i (\omega) - \mu)^2 \stackrel{P}{\rightarrow} \sigma^2 \iff \\ \lim_{N \to \infty} P(\{\omega \in \Omega : |\overline{(X_i (\omega) - \mu)^2} - \sigma^2 | > \epsilon) = 0 \ \forall \ \epsilon > 0 $$ where I've used the bar notation for average. This only tells us that for "most" values of $\omega$ that $\overline{(X_i (\omega) - \mu)^2} - \sigma^2$. We cannot conclude that $\lim \overline{(X_i (\omega) - \mu)^2}$ is a constant, because we have not proved that it equals $\sigma^2$ for every value of $\omega$, just "most."

$\endgroup$
14
  • $\begingroup$ as far as I know $\sigma\overset{P}{=}\sigma'$ implies $\sigma=\sigma'$. They're constant, and if they're not equal you would land on a contradiction. And as much as that's the definition, I beg to disagree. The whole point of identifiability is to show that in the presence of large enough sample size the parameters of the true model are retrievable, and that in essence follows from identifiability. What we call a "data point" is technically a random variable so there is not much difference between the likelihood for that and the true distribution. $\endgroup$
    – Cupitor
    Aug 8 '20 at 18:56
  • $\begingroup$ Well, I can explain in more detail if you'd like, but first consider this: if $x \stackrel{P}{=} y$ implied that $x = y$, why would we ever bother writing $x \stackrel{P}{=} y$? Why not just skip writing this and write $x = y$ instead if we can? $\endgroup$ Aug 8 '20 at 23:23
  • 1
    $\begingroup$ @Cupitor I hope I've answered this in the addendum to my answer. The (weak) law of large numbers does not say that $\lim 1/N \sum (X_i - \mu)^2$ is the constant $\sigma^2$. This is still a random variable, i.e. a non-constant function of $\omega$. $\endgroup$ Aug 9 '20 at 11:29
  • 1
    $\begingroup$ It's possible for a normally distributed random variable $X_1$ to be 0, because $p(0) \ne 0$, so the i.i.d. assumption implies that it is also possible for $X_2 = 0$. You can repeat this argument to prove that there is a sequence $X_1, X_2, \dots$ that is always $0$. Call this outcome $\omega$. What is confusing is that this is a probability 0 event, but that does not mean that this is impossible. $\endgroup$ Aug 12 '20 at 14:31
  • 1
    $\begingroup$ Technically, the outcome is $\omega$ and the event is the set $\{ \omega \}$. I should have been a little clearer. In particular, the existence of a single $\omega$ where the sample variance does not converge to $\sigma^2$ is enough to prevent you from claiming that $\sigma = \sigma '$. $\endgroup$ Aug 12 '20 at 21:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.