What's a real-world example of "overfitting"? I kind of understand what "overfitting" means, but I need help as to how to come up with a real-world example that applies to overfitting. 
 A: My favorite was the Matlab example of US census population versus time:


*

*A linear model is pretty good

*A quadratic model is closer

*A quartic model predicts total annihilation starting next year


(At least I sincerely hope this is an example of overfitting)
http://www.mathworks.com/help/curvefit/examples/polynomial-curve-fitting.html#zmw57dd0e115
A: The study of Chen et al. (2013) fits two cubics to a supposed discontinuity in life expectancy as a function of latitude. 
Chen Y., Ebenstein, A., Greenstone, M., and Li, H. 2013. Evidence on the impact of sustained
exposure to air pollution on life expectancy from China's Huai River policy. Proceedings of the National Academy of Sciences 110: 12936–12941.  abstract
Despite its publication in an outstanding journal, etc., its tacit endorsement by distinguished people, etc., I would still present this as a prima facie example of over-fitting.
A tell-tale sign is the implausibility of cubics. Fitting a cubic implicitly assumes there is some reason why life expectancy would vary as a third-degree polynomial of the latitude where you live.  That seems rather implausible: it is not easy to imagine a plausible physical mechanism that would cause such an effect.
See also the following blog post for a more detailed analysis of this paper: Evidence on the impact of sustained use of polynomial regression on causal inference (a claim that coal heating is reducing lifespan by 5 years for half a billion people).
A: A form of overfitting is fairly common in sports, namely to identify patterns to explain past results by factors that have no or at best vague power to predict future results. A common feature of these "patterns" is that they are often based on very few cases so that pure chance is probably the most plausible explanation for the pattern. 
Examples include things like (the "quotes" are made up by me, but often look similar)

Team A has won all X games since the coach has starting wearing his magical red jacket.

Similar: 

We shall not be shaving ourselves during the playoffs, because that has helped us win the past X games.

Less superstitious, but a form of overfitting as well:

Borussia Dortmund has never lost a Champions League home game to a Spanish opponent when they have lost the previous Bundesliga away game by more than two goals, having scored at least once themselves.

Similar:

Roger Federer has won all his Davis Cup appearances to European opponents when he had at least reached the semi-finals in that year's Australian Open.

The first two are fairly obvious nonsense (at least to me). The last two examples may perfectly well hold true in sample (i.e., in the past), but I would be most happy to bet against an opponent who would let this "information" substantially affect his odds for Dortmund beating Madrid if they lost 4:1 at Schalke on the previous Saturday or Federer beating Djokovic, even if he won the Australian Open that year.
A: When I was trying to understand this myself, I started thinking in terms of analogies with describing real objects, so I guess it's as "real world" as you can get, if you want to understand the general idea:
Say you want to describe to someone the concept of a chair, so that they get a conceptual model that allows them to predict if a new object they find is a chair. You go to Ikea and get a sample of chairs, and start describing them by using two variables: it's an object with 4 legs where you can sit. Well, that may also describe a stool or a bed or a lot of other things. Your model is underfitting, just as if you were to try and model a complex distribution with too few variables - a lot of non-chair things will be identified as chairs. 
So, let's increase the number of variables, add that the object has to have a back, for example. Now you have a pretty acceptable model that describes your set of chairs, but is general enough to allow a new object to be identified as one. Your model describes the data, and is able to make predictions. However, say you happen to have got a set where all chairs are black or white, and made of wood. You decide to include those variables in your model, and suddenly it won't identify a plastic yellow chair as a chair. So, you've overfitted your model, you have included features of your dataset as if they were features of chairs in general, (if you prefer, you have identified "noise" as "signal", by interpreting random variation from your sample as a feature of the whole "real world chairs"). So, you either increase your sample and hope to include some new material and colors, or decrease the number of variables in your models.
This may be a simplistic analogy and breakdown under further scrutiny, but I think it works as a general conceptualization... Let me know if some part needs clarification.
A: In predictive modeling, the idea is to use the data at hand to discover the trends that exist and that can be generalized to future data. By including variables in your model that have some minor, non-significant effect you are abandoning this idea. What you are doing is considering the specific trends in your specific sample that are only there because of random noise instead of a true, underlying trend. In other words, a model with too many variables fits the noise rather than discovering the signal.
Here's an exaggerated illustration of what I'm talking about. Here the dots are the observed data and the line is our model. Look at that a perfect fit - what a great model! But did we really discover the trend or are we just fitting to the noise? Likely the latter. 

A: In a March 14, 2014 article in Science, David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani identified problems in Google Flu Trends that they attribute to overfitting.

Here is how they tell the story, including their explanation of the nature of the overfitting and why it caused the algorithm to fail:

In February 2013, ...
Nature reported that GFT was predicting
more than double the proportion
of doctor visits for influenza-like illness (ILI) than the Centers
for Disease Control and Prevention
(CDC) ... . This happened despite the fact
that GFT was built to predict CDC
reports.
...
Essentially, the methodology
was to find the best matches among 50 million
search terms to fit 1152 data points. The odds of finding search terms that
match the propensity of the flu but are structurally
unrelated, and so do not predict the
future, were quite high. GFT developers,
in fact, report weeding out seasonal search
terms unrelated to the flu but strongly correlated
to the CDC data, such as those regarding
high school basketball. This should
have been a warning that the big data were
overfitting the small number of cases—a
standard concern in data analysis. This ad
hoc method of throwing out peculiar search
terms failed when GFT completely missed
the nonseasonal 2009 influenza A–H1N1
pandemic.

[Emphasis added.]
A: I saw this image a few weeks ago and thought it was rather relevant to the question at hand. 

Instead of linearly fitting the sequence, it was fitted with a quartic polynomial, which had perfect fit, but resulted in a clearly ridiculous answer. 
A: Here is a "real world" example not in the sense that somebody happened to come across it in research, but in the sense that it uses everyday concepts without many statistic-specific terms. Maybe this way of saying it will be more helpful for some people whose training is in other fields. 
Imagine that you have a database with data about patients with a rare disease. You are a medical graduate student and want to see if you can recognize risk factors for this disease. There have been 8 cases of the disease in this hospital, and you have recorded 100 random pieces of information about them: age, race, birth order, have they had measles as a child, whatever. You also have recorded the data for 8 patients without this disease. 
You decide to use the following heuristic for risk factors: if a factor takes a given value in more than one of your diseased patients, but in 0 of your controls, you will consider it a risk factor. (In real life, you'd use a better method, but I want to keep it simple). You find out that 6 of your patients are vegetarians (but none of the controls is vegetarian), 3 have Swedish ancestors, and two of them have a stuttering speech impairment. Out of the other 97 factors, there is nothing which occurs in more than one patient, but is not present among the controls. 
Years later, somebody else takes interest in this orphan disease and replicates your research. Because he works at a larger hospital, which has a data-sharing cooperation with other hospitals, he can use data about 106 cases, as opposed to your 8 cases. And he finds out that the prevalence of stutterers is the same in the patient group and the control group; stuttering is not a risk factor. 
What happened here is that your small group had 25% stutterers by random chance. Your heuristic had no way of knowing if this is medically relevant or not. You gave it criteria to decide when you consider a pattern in the data "interesting" enough to be included in the model, and according to these criteria, the stuttering was interesting enough. 
Your model has been overfitted, because it mistakenly included a parameter which is not really relevant in the real world. It fits your sample - the 8 patients + 8 controls - very well, but it does not fit the real world data. When a model describes your sample better than it describes reality, it's called overfitted. 
Had you chosen a threshold of 3 out of 8 patients having a feature, it wouldn't have happened - but you'd had a higher chance to miss something actually interesting. Especially in medicine, where many diseases only happen in a small fraction of people exhibiting in risk factor, that's a hard trade-off to make. And there are methods to avoid it (basically, compare to a second sample and see if the explaining power stays the same or falls), but this is a topic for another question. 
A: Here's a real-life example of overfitting that I helped perpetrate and then tried (unsuccessfully) to avert:
I had several thousand independent, bivariate time series, each with no more than 50 data points, and the modeling project involved fitting a vector autoregression (VAR) to each one. No attempt was made to regularize across observations, estimate variance components, or anything like that. The time points were measured over the course of a single year, so the data were subject to all kinds of seasonal and cyclical effects that only appeared once in each time series.
One subset of the data exhibited an implausibly high rate of Granger causality compared to the rest of the data. Spot checks revealed that positive spikes were occurring one or two lags apart in this subset, but it was clear from the context that both spikes were caused directly by an external source and that one spike was not causing the other. Out-of-sample forecasts using this models would probably be quite wrong, because the models were overfitted: rather than "smoothing out" the spikes by averaging them into the rest of the data, there were few enough observations that the spikes were actually driving the estimates.
Overall, I don't think the project went badly but I don't think it produced results that were anywhere near as useful as they could have been. Part of the reason for this is that the many-independent-VARs procedure, even with just one or two lags, was having a hard time distinguishing between data and noise, and so was fitting to the latter at the expense of providing insight about the former.
A: My favourite is the “3964 formula” discovered before the World Cup soccer competition in 1998: 
Brazil won the championships in 1970 and 1994. Sum up these 2 numbers  and you will get 3964; Germany won in 1974 and 1990, adding up again to 3964; the same thing with Argentina winning in 1978 and 1986 (1978+1986 = 3964). 
This is a very surprising fact, but everyone can see that it is not advisable to base any future prediction on that rule. And indeed, the rule gives that the winner of the World Cup in 1998 should have been England since 1966 + 1998 = 3964 and England won in 1966.
This didn’t happen and the winner was France.
A: To me the best example is Ptolemaic system in astronomy. Ptolemy assumed that Earth is at the center of the universe, and created a sophisticated system of nested circular orbits, which would explain movements of object on the sky pretty well. Astronomers had to keep adding circles to explain deviation, until one day it got so convoluted that folks started doubting it. That's when Copernicus came up with a more realistic model.
This is the best example of overfitting to me. You can't overfit data generating process (DGP) to the data. You can only overfit misspecified model. Almost all our models in social sciences are misspecified, so the key is to remember this, and keep them parsimonious. Not to try to catch every aspect of the data set, but try to capture the essential features through simplification.
A: Let's say you have 100 dots on a graph.
You could say: hmm, I want to predict the next one.


*

*with a line

*with a 2nd order polynomial

*with a 3rd order polynomial

*...

*with a 100th order polynomial


Here you can see a simplified illustration for this example:

The higher the polynomial order, the better it will fit the existing dots. 
However, the high order polynomials, despite looking like to be better models for the dots, are actually overfitting them. It models the noise rather than the true data distribution. 
As a consequence, if you add a new dot to the graph with your perfectly fitting curve, it'll probably be further away from the curve than if you used a simpler low order polynomial.
A: The analysis that may have contributed to the Fukushima disaster is an example of overfitting. There is a well known relationship in Earth Science that describes the probability of earthquakes of a certain size, given the observed frequency of "lesser" earthquakes. This is known as the Gutenberg-Richter relationship, and it provides a straight-line log fit over many decades. Analysis of the earthquake risk in the vicinity of the reactor (this diagram from Nate Silver's excellent book "The Signal and the Noise") show a "kink" in the data. Ignoring the kink leads to an estimate of the annualized risk of a magnitude 9 earthquake as about 1 in 300 - definitely something to prepare for. However, overfitting a dual slope line (as was apparently done during the initial risk assessment for the reactors) reduces the risk prediction to about 1 in 13,000 years. One could not fault the engineers for not designing the reactors to withstand such an unlikely event - but one should definitely fault the statisticians who overfitted (and then extrapolated) the data...

A: Studying for an exam by memorising the answers to last year's exam.  
A: "Agh! Pat is leaving the company. How are we ever going to find a replacement?"
Job Posting:
Wanted: Electrical Engineer. 
42 year old androgynous person with degrees in Electrical Engineering, mathematics, and animal husbandry.  Must be 68 inches tall with brown hair, a mole over the left eye, and prone to long winded diatribes against geese and misuse of the word 'counsel'.
In a mathematical sense, overfitting often refers to making a model with more parameters than are necessary, resulting in a better fit for a specific data set, but without capturing relevant details necessary to fit other data sets from the class of interest.
In the above example, the poster is unable to differentiate the relevant from irrelevant characteristics.  The resulting qualifications are likely only met by the one person that they already know is right for the job (but no longer wants it).
A: Here's a nice example of presidential election time series models from xkcd:

There have only been 56 presidential elections and 43 presidents. That is not a lot of data to learn from. When the predictor space expands to include things like having false teeth and the Scrabble point value of names, it's pretty easy for the model to go from fitting the generalizable features of the data (the signal) and to start matching the noise. When this happens, the fit on the historical data may improve, but the model will fail miserably when used to make inferences about future presidential elections.
A: Many intelligent people in this thread --- many much more versed in statistics than I am. But I still don't see an easy-to-understand to the lay-person example. The Presidential example doesn't quite hit the bill in terms of typical overfitting, because while it is technically overfitting in each one of its wild claims, usually an overfitting model overfits -ALL- the given noise, not just one element of it.
I really like the chart in the bias-variance tradeoff explanation in wikipedia:
http://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff
(The lowermost chart is the example of overfitting).
I'm hard pressed to think of a real world example that doesn't sound like complete mumbo-jumbo. The idea is that data is part caused by measurable, understandable variables --- part random noise. Attempting to model this noise as a pattern gives you inaccuracy.
A classic example is modeling based SOLELY on R^2 in MS Excel (you are attempting to fit an equation/ model literally as close as possible to the data using polynomials, no matter how nonsensical).
Say you're trying to model ice cream sales as a function of temperature. You have "real world" data. You plot the data and try to maximize R^2. You'll find using real-world data, the closest fit equation is not linear or quadratic (which would make logical sense). Like almost all equations, the more nonsensical polynomial terms you add (x^6 -2x^5 +3x^4+30x^3-43.2x^2-29x) -- the closer it fits the data. So how does that sensibly relate temperature to ice cream sales? How would you explain that ridiculous polynomial? Truth is, it's not the true model. You've overfit the data.
You are taking unaccounted for noise -- which may have been due to sales promotions or some other variable or "noise" like a butterfly flapping its wings in the cosmos (something never predictable)--- and attempted to model that based on temperature. Now usually if your noise/ error does not average to zero or is auto-correlated, etc, it means there are more variables out there --- and then eventually you get to generally randomly distributed noise, but still, that's the best I can explain it.
A: Most optimization methods have some fudge factors aka hyperparameters.
A real example:

For all systems under study, the following parameters yielded a fast and robust behavior:
  $N_{min} = 5,\ \ f_{inc} = 1.1,\ \ f_{dec} = 0.5,\ \ \alpha_{start} = 0.1, \ \ f_{\alpha} = 0.99.$  

Is this over fitting, or just fitting to a particular set of problems ?
A: A bit intuitive, but maybe it'll help. Let's say you want to learn some new language. How do you learn? instead of learning the rules in a course, you use examples. Specifically, TV shows. So you like crime shows, and you watch few series of some cop show. Then, you take another crime show and watch some series form that one. By the third show you see - you know almost everything, no problem. You don't need the English subtitles.
But then you try your newly learned language on the street on your next visit, and you realize you can't talk about anything other than saying "officer! that man took my bag and shot that lady!". While your 'training error' was zero, your 'test error' is high, due to 'overfitting' the language, studying only a limited subset of words and assuming its enough.
