1
$\begingroup$

This question is inspired by this model correlation plot from the Kaggle Zillow Competition, specifically Zillow's blog summary:

enter image description here

The competition was to make a model that predicts house prices using historic pricing data and various house features. The correlation plot above compares the top performing models on the same houses. In particular, the blog writes:

"The top two teams on the leaderboard, Zensemble and Juan Zhai, had solutions that were highly correlated suggesting that their modeling took similar approaches. Indeed, we later found out that this was the case. But, the next two teams, Silogram and Jack from Japan, had solutions that were weakly correlated with the top two solutions and each other! This means that among the top teams there were many different ways to improve on the Zestimate and that we should (and did) talk to them all to get an understanding of their approaches. Combining the highly uncorrelated solutions together is likely going to lead to a larger improvement in the Zestimate than taking improvements from the top teams solution alone."

How effective is the above statement?

Here's what I mean. The general trend in the machine learning community is to write papers about new models and techniques, and then present a table comparing the proposed model to other state-of-the-art approaches, and then compare error rates. Occasionally, authors go further to demonstrate the model outperforms others in particular subsets of the data. This tends to be followed by mostly heuristic arguments for why the model works better than others.

I wonder though, if we could glean more information about differences in models and approaches through such correlation plots (before diving into differences on specific samples)?

Related to the above, if you have two models $f,g$ and some nice convex loss function $L[h](x_i,y_i)$, where $h=f,g$, then what is the relationship between the correlation on the test set $\mbox{Corr}_{test}(f,g)$ and the performance (loss) of $af(x)+bg(x)$, i.e. a mixture of the two models? I'm hoping this should be some standard result in statistical machine learning.

$\endgroup$
5
  • $\begingroup$ What do you mean by "how effective is the above statement?" Do you mean "consistent" or "sensible"? More precisely: Is your question whether Zillow's observation is in fact possible? $\endgroup$
    – AdamO
    Jul 11, 2018 at 18:37
  • $\begingroup$ @AdamO: I know my question is a little vague, and I'm hoping to improve it through some feedback. It's just that I've never really seen this model correlation technique used to compare models, and I'm wondering why. I mean, if you're comparing two really good models on some dataset, and you find their correlation is low, how useful is that observation toward building a better model? I'm not asking if it's possible as it clearly is on a toy example with two data points $x_1,x_2$ where model $A$ gets data point $x_1$ correct, $x_2$ wrong and vice versa for model $B$. $\endgroup$
    – Alex R.
    Jul 11, 2018 at 18:48
  • $\begingroup$ Said another way, how useful of a measure is model correlation for answering "how different are these two models in terms of what they've learned?" In contrast to looking at differences on specific samples, or categories of samples. The second question is mainly toward understanding which pair of models to combine to get a better model. $\endgroup$
    – Alex R.
    Jul 11, 2018 at 18:48
  • $\begingroup$ All the Zillow analysts are saying is: models whose outputs are correlated used similar approaches. $\endgroup$
    – AdamO
    Jul 11, 2018 at 19:01
  • $\begingroup$ To your broader question: mashing predictive models can be done. The most popular is Bayesian Model Averaging. It does exactly what you show in the display. The difference is that the weights can be trained from data. The cost of the precision? It's a complicated model, hard to validate, and hard to implement. $\endgroup$
    – AdamO
    Jul 11, 2018 at 19:20

1 Answer 1

1
$\begingroup$

"... This means that among the top teams there were many different ways to improve on the Zestimate and that we should (and did) talk to them all to get an understanding of their approaches. Combining the highly uncorrelated solutions together is likely going to lead to a larger improvement in the Zestimate than taking improvements from the top teams solution alone."

How effective is the above statement?

Here's what I mean. The general trend in the machine learning community is to write papers about new models and techniques, and then present a table comparing the proposed model to other state-of-the-art approaches, and then compare error rates.

It's a sound method but note that Zillow claims that their own results exceed those of the competition's contestants, and that the top 20 were all very close (way within any margin of error).

Photo Finish

It was a close contest all the way to the end, the difference between first place and 10th place was less than 0.5%. Even smaller differences separated many teams in the top 20 as the chart below shows. Interestingly, the top five or so teams had bigger separation in performance than teams ten to twenty which suggests that superior modeling skills do rise to the top with these type of contests. Also, note the sharp drop between 11th and 12th places – it looks like each of these teams found some signal that eluded the other four thousand folks.

Zestimate vs. Contestants

Zillow reports their results for their Zestimate (a proprietary algorithm) as follows:

Median Error:

Half of the Zestimates in an area were closer than the error percentage and half were farther off. The median error rate for the country is currently 4.6%, meaning half of Zestimates nationwide were within 4.6% of the final selling price, and half are off by more than 4.6%.

Zillow claims that the median home price in the U.S. is "Zillow Home Value Index \$216,000". Note: \$216,000 * 4.6% = \$9,936 and they also claim: "within 20% of the final sale price 85.8% of the time".

Zillow Accuracy

The Zestimate’s accuracy depends on location and availability of data in an area. Some counties have deeply detailed information on homes such as number of bedrooms, bathrooms and square footage and others do not. The more data available, the more accurate the Zestimate value.

Our estimating method differs from that of a comparative market analysis (CMA) done by real estate agents. Geographically, the data we use is much larger than your neighborhood. Often times, we use all the data in a county for calculation.

See their "Data Coverage and Zestimate Accuracy Table" for more details.

How do we come up with the Zestimate and what's in the formula?

We use proprietary automated valuation models that apply advanced algorithms to analyze our data to identify relationships within a specific geographic area, between this home-related data and actual sales prices. Home characteristics, such as square footage, location or the number of bathrooms, are given different weights according to their influence on home sale prices in each specific geography over a specific period of time, resulting in a set of valuation rules, or models that are applied to generate each home's Zestimate.

Notice that it doesn't include factors that might be important to some people: How far away is shopping, the New England Patriots stadium, the ocean, or what's the average weather forecast.

Can I use the Zestimate to get a loan?

No, you can't. To get a federally guaranteed loan, a law called FIRREA (the Federal Institutions Reform, Recovery and Enforcement Act) requires an appraisal from a professional appraiser. Without limitation, lending professionals and institutions are prohibited from using the services in making any loan-related decisions.

We encourage buyers, sellers, and homeowners to supplement Zillow's information by doing other research such as:

  • Getting a comparative market analysis (CMA) from a real estate agent
  • Getting an appraisal from a professional appraiser
  • Visiting the house (whenever possible)

Zillow also produces a Zestimate forecast, which is Zillow’s prediction of a home’s Zestimate one year from now, based on current home and market information.

I think if I was interested in a Zestimate that I'd want the forecast too.

I wonder though, if we could glean more information about differences in models and approaches through such correlation plots (before diving into differences on specific samples)?

Since Zillow's algorithm is proprietary and claimed to be much better it's likely that they will study the contestants submissions to improve their own methods without publishing any details.

In general, cross-validating different models to determine which provides the closest answer to what you expect (or historical data) is possible; predicting the future accurately is a continuing area of research.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.