Why instrumental variables? Or: why not nonparametric regression? Usually instrumental variables are introduced as a means to solve the problem $E(u|X)\neq 0$ in the model $Y = X'\beta + u$. This may happen if we omit important variables from the covariate vector $X$, for instance. However, it is always the case that we can write $Y = E(Y|X) + v$ with $E(v|X)=0$. Moreover, $E(Y|X)$ is always of the form $g\circ X$ for some measurable function $g$. That is, the model $Y = g\circ X + v$ with $E(v|X)=0$ is always correctly specified (even though we may not know what $g$ looks like). Omitted variables play no role here, as it is a probabilistic fact. Thus the problem is not really that we have omitted `important' variables or that $E(Y - X'\beta\, |\, X)\neq 0$, but rather that we are too narrow minded in demanding that $g$ be a linear function! Now, estimating the function $g$ is precisely the aim of nonparametric regression methods. My question is as follows: is there any reason (aside from tradition) to stick with the instrumental variables framework (which is in my view very clumsy) in lieu of adopting nonparametric regression models to describe relationships between variables? (This is a honest question in spite of its seemingly provocative tone)
 A: IV methods are still useful as nonlinearity is by far not the only important way in which misspecification may arise, and hence estimating $g$ is not always what we aim for. Basically, you mention it yourself when writing that there may be omitted variables and that $u\neq v$. 
Let me try a time-honored economic example, "returns to schooling":
There is probably a positive univariate relationship between earnings and the years of schooling you received (it is also probably nonlinear, as adding a PhD to an MSc probably does not have the same marginal benefit as adding a BSc to a high school degree, but that is not important for the argument). You could then estimate this conditional expectation $g$ by nonparametric methods (see, e.g., here), which would tell you how much more an average, say, MSc graduate makes than an average BSc graduate. 
This is, however, very likely not the causal effect of the master's degree (i.e., the difference in earnings is not solely due to the MSc degree) as different types of people choose to receive or not an additional degree. One may expect MSc graduates to be (on average!) more intelligent (which, for example, makes the idea of having to sit more exams relatively less unappealing), to have more perseverance etc. than BSc graduates.
These are also qualifications that will be useful in a later career, hence boost earnings. Hence, these MSc graduates will later not only earn more because they learned useful stuff during their MSc, but also simply because they are who they are. 
Hence, $E(u|\text{yrs of schooling})\neq0$, where $u$ represents things like ability: as ability is not part of the set of regressors, it is part of the error term, and, as discussed above, we expect (i.e., on average) more highly schooled employees to also be more intelligent. 
Instruments may help us get out of this problem.
