Possibility of solution in overdetermined system of moment conditions Hayashi, in page 207-208 of his book Econometrics, ex.3 (see hint), discusses the possibility that when referring to the moment conditions that will determine the estimator formula, having an overidentified system of such conditions. He says that even though there is no solution for the sample overidentified system, there can still be a solution for the population overidentified system. I do not understand how it's possible...
Any help would be appreciated.
P.S: Also posted the same question here
 A: I'll adopt Hayashi's notation. We have $K$ estimating equations of the form:
$$g(w_i;\delta)=y_i-z_i^\prime\delta$$
where $\delta$ is a parameter vector of length $L$ and $y_i=z_i^\prime\delta +\sigma_i$. 
We know $E[g(w_i; \delta)]=0$. To estimate these equations we use the solution to a linear set of equations:
$$E[x_iy_i]-E[x_iz_i^\prime]\hat\delta = 0\,\text{  or  }\,\Sigma_{xz}\hat\delta=\sigma_{xy}$$
The population analog of $E[g(w_i;\hat\delta)]$ is $g_n(\hat\delta)\equiv \dfrac{1}{n}\sum_{i=1}^ng(w_i;\hat\delta)$, and the population version of this system of equations would be:
\begin{equation}
S_{xz}\hat\delta=s_{xy}
\end{equation}
$$\text{where: } s_{xy}\equiv\dfrac{1}{n}\sum_{i=1}^nx_iy_i\, \text{  and  }\, S_{xz}\equiv \dfrac{1}{n}\sum_{i=1}^n x_i z_i^\prime$$
There are two parts of this proof, much of it is somewhat tautological.
1. The population version has a solution
Rouché Capelli theorem is going to be our workhorse. It states:

A system of linear equations $Ax=b$ has a solution if and only if the rank of its coefficient matrix $A$ is equal to the rank of its augmented matrix $[A|b]$.

One of the identifying assumptions of GMM is that $\Sigma_{xz}$ has rank $L$. So we need to show that $[\Sigma_{xz} | \sigma_{xy}]$ has rank $L$. To do this we have to show there is a linear combination of rows such that $\Sigma_{xz}\beta + \sigma_{xy}=0$. But by assumption we know that for $\beta=\delta$ this is true. So the rank of $[\Sigma_{xz} | \sigma_{xy}]$ which means there must be a solution to this set of equations.
Note that this part works by assumption. We know that for the entire distribution $\delta$ is a solution to this set of equations, so a solution to this set of equations must exist.
2. The sample version does not have a solution
Again, we're going to use the Rouché Capelli theorem. We now want to show that the rank of $[S_{xz}|s_{xy}]$ is $L+1$, even for $n$ extremely large.
In order to eliminate $s_{xy}$ we need $g_n(\delta)$ to be zero for large enough $n$. Whether we can get this is really a question of what the law of large numbers buys us. We know that:
$$g_n(\delta)\xrightarrow{a. s.}0$$
which really corresponds to saying that given any $\epsilon>0$ we can pick an $N$ such that:
$$||g_n|| < \epsilon \text{ with probability 1 }$$ 
for all $n>N$.
However we need $g_n(\delta)=0$ exactly in order to eliminate $s_{xy}$. So unless we have the full population, $S_{xz}$ will have rank $L+1$.
Meanwhile the condition for $S_{xz}$ to have full column rank is simply that there is no $\gamma\neq 0$ such that $S_{xz}\gamma = 0$. Take any $\gamma$. There is some $\Gamma\neq 0$ such that $\Gamma = \Sigma_{xz}\gamma$, since $\Sigma_{xz}$ has full column rank. Then by the LLN there is an $N$ large enough such that for any $n>N$, $S_{xz}\gamma = \Gamma + \epsilon \neq 0$ with probability 1. So for $n>N$, $S_{xz}$ has rank $L$. Therefore the system will not have a solution.
To summarize: we know that the system of equations have a solution in the population because that's precisely our underlying assumption for GMM. But for any random sample, no matter how large, the system will not have a solution because minute random variations around the population mean prevent those equations from holding exactly.
