"... This means that among the top teams there were many different ways to improve on the Zestimate and that we should (and did) talk to them all to get an understanding of their approaches. Combining the highly uncorrelated solutions together is likely going to lead to a larger improvement in the Zestimate than taking improvements from the top teams solution alone."
How effective is the above statement?
Here's what I mean. The general trend in the machine learning community is to write papers about new models and techniques, and then present a table comparing the proposed model to other state-of-the-art approaches, and then compare error rates.
It's a sound method but note that Zillow claims that their own results exceed those of the competition's contestants, and that the top 20 were all very close (way within any margin of error).
It was a close contest all the way to the end, the difference between first place and 10th place was less than 0.5%. Even smaller differences separated many teams in the top 20 as the chart below shows. Interestingly, the top five or so teams had bigger separation in performance than teams ten to twenty which suggests that superior modeling skills do rise to the top with these type of contests. Also, note the sharp drop between 11th and 12th places – it looks like each of these teams found some signal that eluded the other four thousand folks.
Zillow reports their results for their Zestimate (a proprietary algorithm) as follows:
Half of the Zestimates in an area were closer than the error percentage and half were farther off. The median error rate for the country is currently 4.6%, meaning half of Zestimates nationwide were within 4.6% of the final selling price, and half are off by more than 4.6%.
Zillow claims that the median home price in the U.S. is "Zillow Home Value Index \$216,000". Note: \$216,000 * 4.6% = \$9,936 and they also claim: "within 20% of the final sale price 85.8% of the time".
The Zestimate’s accuracy depends on location and availability of data in an area. Some counties have deeply detailed information on homes such as number of bedrooms, bathrooms and square footage and others do not. The more data available, the more accurate the Zestimate value.
Our estimating method differs from that of a comparative market analysis (CMA) done by real estate agents. Geographically, the data we use is much larger than your neighborhood. Often times, we use all the data in a county for calculation.
See their "Data Coverage and Zestimate Accuracy Table" for more details.
How do we come up with the Zestimate and what's in the formula?
We use proprietary automated valuation models that apply advanced algorithms to analyze our data to identify relationships within a specific geographic area, between this home-related data and actual sales prices. Home characteristics, such as square footage, location or the number of bathrooms, are given different weights according to their influence on home sale prices in each specific geography over a specific period of time, resulting in a set of valuation rules, or models that are applied to generate each home's Zestimate.
Notice that it doesn't include factors that might be important to some people: How far away is shopping, the New England Patriots stadium, the ocean, or what's the average weather forecast.
Can I use the Zestimate to get a loan?
No, you can't. To get a federally guaranteed loan, a law called FIRREA (the Federal Institutions Reform, Recovery and Enforcement Act) requires an appraisal from a professional appraiser. Without limitation, lending professionals and institutions are prohibited from using the services in making any loan-related decisions.
We encourage buyers, sellers, and homeowners to supplement Zillow's information by doing other research such as:
- Getting a comparative market analysis (CMA) from a real estate agent
- Getting an appraisal from a professional appraiser
- Visiting the house (whenever possible)
Zillow also produces a Zestimate forecast, which is Zillow’s prediction of a home’s Zestimate one year from now, based on current home and market information.
I think if I was interested in a Zestimate that I'd want the forecast too.
I wonder though, if we could glean more information about differences in models and approaches through such correlation plots (before diving into differences on specific samples)?
Since Zillow's algorithm is proprietary and claimed to be much better it's likely that they will study the contestants submissions to improve their own methods without publishing any details.
In general, cross-validating different models to determine which provides the closest answer to what you expect (or historical data) is possible; predicting the future accurately is a continuing area of research.