Proving Identifiability Using Law of Large Numbers? Well normally proving identifiability follows by showing that $p_{\theta}(x)=p_{\theta'}(x)$ implies $\theta=\theta'$. Usually this proceeds by showing that a function dependent on $\theta$, such as log-likelihood (or conditional log-likelihood in the case of Logistic Regression) leading to equality implies the equality  $\theta=\theta'$. In that sense it is like showing the likelihood function is one-to-one when applied to a single data point. To be more precise for a dataset of size one $\mathcal{D}=\{X\}$ from Gaussian distribution, to show identifiability we write:
$$nll_{\mu, \sigma}(\mathcal{D})=c + \log(\sigma) + \frac{1}{2{\sigma}^2}(X-\mu)^2=c + \log(\sigma') + \frac{1}{2{\sigma'}^2}(X-\mu')^2$$. Now taking the derivative twice with respect to $X$ we get
$$\frac{1}{\sigma^2}=\frac{1}{{\sigma'}^2}$$
which then implies $\sigma=\sigma'$. Conclusion about $\mu=\mu'$ also follows with similar algebraic manipulations.
My question is what if we imply identifiablity for the likelihood of a large set of points rather than one point only, and in doing so, we could incorporate limiting laws such as LLN. For example in the case of identifiability of Gaussian distribution we can do as follows. Again assume  $X_i\sim \mathcal{N}(\mu, \sigma)$.
Now if we have
$$
nll_\mu(\mathcal{D})=c + N\log(\sigma) + \frac{1}{2{\sigma}^2}\sum_{i=1}(X_i-\mu)^2=c +N\log(\sigma')+ \frac{1}{2{\sigma'}^2}\sum_{i=1}(X_i-\mu')^2
=nll_{\mu'}(\mathcal{D})$$
dividing both sides with $N$ and taking the limit of $N\to\infty$ which implies using SLLN:
$$\log(\sigma)+\frac{1}{2}\overset{P}{=} \log(\sigma')+\frac{1}{2}$$
which implies $\sigma=\sigma'$.
So rather than using a single point, I am using a large sample of points to show identifiability. Is this allowed? If so where can I find this formally written in a textbook? If not, why not? Isn't the point to show that when large enough sample is gathered we can identify the true parameter anyway?
 A: Ultimately, the problem here is that $\sigma \stackrel{P}{=} \sigma' $ does not imply that $\sigma = \sigma'$, so no, you cannot. A couple of notes though:

Well normally proving identifiability follows by showing that $p_{\theta}(x)=p_{\theta'}(x)$ implies $\theta=\theta'$.

Since the definition of identifiability is that the function $\theta \mapsto p_\theta$ is injective (one-to-one), every proof must ultimately show that $p_{\theta}(x)=p_{\theta'}(x)$ implies $\theta=\theta'$, since that is the definition of an injective function. There might be different ways to arrive at this conclusion, but it is a necessity for the proof.

[this] is like showing the likelihood function is one-to-one when applied to a single data point.

This is true, but nothing about the function $\theta \mapsto p_\theta$ has anything to do with data points, it's simply the case that the two look the same analytically when there is one data point. The function $\theta \mapsto p_\theta$ is determined only by how we decided to represent a certain set of distributions by their parameters. It's all a priori and not based on any kind of data collection whatsoever.
Addendum: Okay, let's go through it formally. Let $\mathscr{L}$ denote the negative log-likelihood, which we need to view as a function not only of the parameters $\mu$ and $\sigma$, but also as a random variable, i.e. as a function of $\omega \in \Omega$ where $\Omega$ is the sample space:
$$
\mathscr{L}_{\mu , \sigma} (\omega) = c + N \log(\sigma) + \frac{1}{2\sigma^2} \sum_{i=1}^N (X_i (\omega) - \mu)^2
$$
Start with the assumption that $p_{\mu, \sigma} = p_{\mu', \sigma'}$. Since we have two distributions, we should have two samples, say $X_i$ and $Y_i$, and we know that $E(X_i) = \mu$, $V(X_i) = \sigma^2$, $E(Y_i) = \mu'$, and $V(Y_i) = \sigma'^2$. We can cancel the constants $c$ and write
\begin{align}
N \log(\sigma) + \frac{1}{2\sigma^2} \sum_{i=1}^N (X_i (\omega) - \mu)^2 & = N \log(\sigma') + \frac{1}{2\sigma'^2} \sum_{i=1}^N (Y_i (\omega) - \mu')^2 \\
N \log(\sigma) + \frac{N}{2\sigma^2} \left(\frac{1}{N}\right) \sum_{i=1}^N (X_i (\omega) - \mu)^2 & = N \log(\sigma') + \frac{N}{2\sigma'^2} \left(\frac{1}{N}\right) \sum_{i=1}^N (Y_i (\omega) - \mu')^2
\end{align}
and now we want to apply the weak law of large numbers (WLLN) (since you are using the notation for convergence in probability, not almost sure convergence like you would get from using the strong law of large numbers) to both sides which tells us that
\begin{align}
\left(\frac{1}{N}\right) \sum_{i=1}^N (X_i (\omega) - \mu)^2 & \stackrel{P}{\rightarrow} E[(X_i (\omega) - \mu)^2] = \sigma^2 \\
\left(\frac{1}{N}\right) \sum_{i=1}^N (Y_i (\omega) - \mu')^2 & \stackrel{P}{\rightarrow} E[(Y_i (\omega) - \mu')^2] = \sigma'^2.
\end{align}
Let's remember what convergence in probability means though:
$$
\left(\frac{1}{N}\right) \sum_{i=1}^N (X_i (\omega) - \mu)^2 \stackrel{P}{\rightarrow} \sigma^2 \iff \\
\lim_{N \to \infty} P(\{\omega \in \Omega : |\overline{(X_i (\omega) - \mu)^2} - \sigma^2 | > \epsilon) = 0 \ \forall \  \epsilon > 0
$$
where I've used the bar notation for average. This only tells us that for "most" values of $\omega$ that $\overline{(X_i (\omega) - \mu)^2} - \sigma^2$. We cannot conclude that $\lim \overline{(X_i (\omega) - \mu)^2}$ is a constant, because we have not proved that it equals $\sigma^2$ for every value of $\omega$, just "most."
