What do "endogeneity" and "exogeneity" mean substantively? I understand that the basic definition of endogeneity is that 
$$
X'\epsilon=0
$$ 
is not satisfied, but what does this mean in a real world sense? I read the Wikipedia article, with the supply and demand example, trying to make sense of it, but it didn't really help. I've heard the other description of endogenous and exogenous as being within the system and being outside the system and that still doesn't make sense to me.
 A: The OLS regression, by construction, gives $X'\epsilon=0$.  Actually that is not correct.  It gives $X'\hat\epsilon=0$ by construction.  Your estimated residuals are uncorrelated with your regressors, but your estimated residuals are "wrong" in a sense.  
If the true data-generating-process operates by $Y=\alpha +\beta X + \gamma Z + {\rm noise}$, and $Z$ is correlated with $X$, then $X'{\rm noise} \neq 0$ if you fit a regression leaving out $Z$.  Of course, the estimated residuals will be uncorrelated with $X$.  They always are, the same way that $\log(e^x)=x$.  It is just a mathematical fact.  This is the omitted variable bias.
Say that $I$ is randomly assigned.  Maybe it is the day of week that people are born.  Maybe it is an actual experiment.  It is anything uncorrelated with $Y$ that predicts $X$.  You can then use the randomness of $I$ to predict $X$, and then use that predicted $X$ to fit a model to $Y$.  
That is two stage least squares, which is almost the same as IV.
A: JohnRos's answer is very good.  In plain English, endogeneity means you got the causation wrong.  That the model you wrote down and estimated does not properly capture the way causation works in the real world.  When you write:
\begin{equation}
Y_i=\beta_0+\beta_1X_i+\epsilon_i
\end{equation}
you can think of this equation in a number of ways.  You could think of it as a convenient way of predicting $Y$ based on $X$'s values.  You could think of it as a convenient way of modeling $E\{Y|X\}$.  In either of these cases, there is no such thing as endogeneity, and you don't need to worry about it.
However, you can also think of the equation as embodying causation.  You can think of $\beta_1$ as the answer to the question: "What would happen to $Y$ if I reached in to this system and experimentally increased $X$ by 1?"  If you want to think about it that way, using OLS to estimate it amounts to assuming that:

*

*$X$ causes $Y$

*$\epsilon$ causes $Y$

*$\epsilon$ does not cause $X$

*$Y$ does not cause $X$

*Nothing which causes $\epsilon$ also causes $X$
Failure of any one of 3-5 will generally result in $E\{\epsilon|X\}\ne0$, or, not quite equivalently, ${\rm Cov}(X,\epsilon)\ne0$.  Instrumental variables is a way of correcting for the fact that you got the causation wrong (by making another, different, causal assumption).  A perfectly conducted randomized controlled trial is a way of forcing 3-5 to be true.  If you pick $X$ randomly, then it sure ain't caused by $Y$, $\epsilon$, or anything else.  So-called "natural experiment" methods are attempts to find special circumstances out in the world where 3-5 are true even when we don't think 3-5 are usually true.
In JohnRos's example, to calculate the wage value of education, you need a causal interpretation of $\beta_1$, but there are good reasons to believe that 3 or 5 is false.
Your confusion is understandable, though.  It is very typical in courses on the linear model for the instructor to use the causal interpretation of $\beta_1$ I gave above while pretending not to be introducing causation, pretending that "it's all just statistics."  It's a cowardly lie, but it's also very common.
In fact, it is part of a larger phenomenon in biomedicine and the social sciences.  It is almost always the case that we are trying to determine the causal effect of $X$ on $Y$---that's what science is about after all.  On the other hand, it is also almost always the case that there is some story you can tell leading to a conclusion that one of 3-5 is false.  So, there is a kind of practiced, fluid, equivocating dishonesty in which we swat away objections by saying that we're just doing associational work and then sneak the causal interpretation back elsewhere (normally in the introduction and conclusion sections of the paper).
If you are really interested, the guy to read is Judea Perl.  James Heckman is also good.
A: Let me use an example:
Say you want to quantify the (causal) effect of education on income. You take education years and income data and regress one against the other. Did you recover what you wanted? Probably not! This is because the income is also caused by things other than education, but which are correlated to education. Let's call them "skill": We can  safely assume that education years are affected by "skill", as the more skilled you are, the easier it is to gain education. So, if you regress education years on income, the estimator for the education effect absorbs the effect of "skill" and you get an overly optimistic estimate of return to education. This is to say, education's effect on income is (upward) biased because education is not exogenous to income.
Endogeneity is only a problem if you want to recover causal effects (unlike mere correlations). Also- if you can design an experiment, you can guarantee that ${\rm Cov}(X,\epsilon)=0$ by random assignment. Sadly, this is typically impossible in social sciences.
A: User25901 is looking for a straight-forward, simple, real-world explanation of what the terms exogenous and endogenous mean.  To respond with arcane examples or mathematical definitions is to not really answer the question that was asked.
How do I, 'get a gut understanding of these two terms?'
Here's what I came up with:
Exo - external, outside
Endo - internal, inside
-genous - originating in
Exogenous: A variable is exogenous to a model if it is not determined by other parameters and variables in the model, but is set externally and any changes to it come from external forces.
Endogenous: A variable is endogenous in a model if it is at least partly function of other parameters and variables in a model.
Therefore, exo-geneity and exo-geneocity is the reflexive adjective of the right-hand-side cause, describing its effect on others, rather than its own integrality or derivations.
A: Think of a system as $x,y$. When we're trying to explain it by a model $y=f(x)+\varepsilon$, is the error $\varepsilon$ a part of the system or not?
When the error is not part of the system, we call it exogenous, i.e. it's added to $f(x)$ after $x$ had its input into the system.
When the error is a part of the system, we call it endogenous, i.e. not only it enters $y$ after $f(x)$, it also enters $x$ itself somehow before $f(.)$ is applied to it.
This makes $endogenous$ models troublesome, for they interfere with our attempts to estimate the function $f(.)$.
A: In regression we want to capture the quantitative impact of an independent variable (which we assume is exogenous and not being itself dependent on something else) on an identified dependent variable. We want to know what net effect an exogenous variable has on a dependent variable- meaning the independent variable should be free of any influence from another variable. A quick way to see if the regression is suffering from the problem of endogeneity is to check the correlation between the independent variable and the residuals. But this is just a rough check otherwise formal tests of endogeneity need to be undertaken. 
