Industry vs Kaggle challenges. Is collecting more observations and having access to more variables more important than fancy modelling?

Question

I'd hope the title is self explanatory. In Kaggle, most winners use stacking with sometimes hundreds of base models, to squeeze a few extra % of MSE, accuracy... In general, in your experience, how important is fancy modelling such as stacking vs simply collecting more data and more features for the data?

Answer 1

By way of background, I have been doing forecasting store $\times$ SKU time series for retail sales for 12 years now. Tens of thousands of time series across hundreds or thousands of stores. I like saying that we have been doing Big Data since before the term became popular.

I have consistently found that the single most important thing is to understand your data. If you don't understand major drivers like Easter or promotions, you are doomed. Often enough, this comes down to understanding the specific business well enough to ask the correct questions and telling known unknowns from unknown unknowns.

Once you understand your data, you need to work to get clean data. I have supervised quite a number of juniors and interns, and the one thing they had never experienced in all their statistics and data science classes was how much sheer crap there can be in the data you have. Then you need to either go back to the source and try to get it to bring forth good data, or try to clean it, or even just throw some stuff away. Changing a running system to yield better data can be surprisingly hard.

Once you understand your data and actually have somewhat-clean data, you can start fiddling with it. Unfortunately, by this time, I have often found myself out of time and resources.

I personally am a big fan of model combination ("stacking"), at least in an abstract sense, less so of fancy feature engineering, which often crosses the line into overfitting territory - and even if your fancier model performs slightly better on average, one often finds that the really bad predictions get worse with a more complex model. This is a dealbreaker in my line of business. A single really bad forecast can pretty completely destroy the trust in the entire system, so robustness is extremely high in my list of priorities. Your mileage may vary.

In my experience, yes, model combination can improve accuracy. However, the really big gains are made with the first two steps: understanding your data, and cleaning it (or getting clean data in the first place).

Answer 2

I can't speak for the whole of industry, obviously, but I work in industry and have competed on Kaggle so I will share my POV.

First, you're right to suspect that Kaggle doesn't exactly match what people are doing in industry. It's a game, and subject to gamesmanship, with lots of crazy restrictions. For example, in the currently running Santander competition:

1. The feature names were artificially hashed to hide their meaning
2. The "training" set was artificially limited to have fewer rows than columns specifically so that feature selection, robustness, and regularization technique would be indispensable to success.
3. The so-called "test" set has a markedly different distribution than the training set and the two are clearly not random samples from the same population.

If someone gave me a data set like this at work, I would immediately offer to work with them on feature engineering so we could get features that were more useful. I would suggest we use domain knowledge to decide on likely interaction terms, thresholds, categorical variable coding strategies, etc. Approaching the problem in that way would clearly be more productive than trying to extract meaning from an exhaust file produced by a database engineer with no training in ML.

Furthermore, if you learn, say, that a particular numeric column is not numeric at all but rather a ZIP code, well, you can go and get data from 3rd-party data sources such as the US Census to augment your data. Or if you have a date, maybe you'll include the S&P 500 closing price for that day. Such external augmentation strategies require detailed knowledge of the specific data set and significant domain knowledge but usually have the much larger payoffs than pure algorithmic improvements.

So, the first big difference between industry and Kaggle is that in industry, features (in the sense of input data) are negotiable.

A second class of differences is performance. Often, models will be deployed to production in one of two ways: 1) model predictions will be pre-computed for every row in a very large database table, or 2) an application or website will pass the model a single row of data and need a prediction returned in real-time. Both use cases require good performance. For these reasons, you don't often see models that can be slow to predict or use a huge amount of memory like K-Nearest-Neighbors or Extra Random Forests. A logistic regression or neural network, in contrast, can score a batch of records with a few matrix multiplications, and matrix multiplication can be highly optimized with the right libraries. Even though I could get maybe +0.001 AUC if I stacked on yet another non-parametric model, I wouldn't because prediction throughput and latency would drop too much.

There's a reliability dimension to this as well - stacking four different state-of-the-art 3rd-party libraries, say LightGBM, xgboost, catboost, and Tensorflow (on GPUs, of course) might get you that .01 reduction in MSE that wins Kaggle competitions, but it's four different libraries to install, deploy, and debug if something goes wrong. It's great if you can get all that stuff working on your laptop, but getting it running inside a Docker container running on AWS is a completely different story. Most companies don't want to front a small devops team just to deal with these kinds of deployment issues.

That said, stacking in itself isn't necessarily a huge deal. In fact, stacking a couple different models that all perform equally well but have very different decision boundaries is a great way to get a small bump in AUC and a big bump in robustness. Just don't go throwing so many kitchen sinks into your heterogeneous ensemble that you start to have deployment issues.

Answer 3

From my experience, more data and more features are more important than the fanciest, most stacked, most tuned, model one can come up with.

Look at the online advertising competitions that took place. Winning models were so complex they ended up taking a whole week to train (on a very small dataset, compared to the industry standard). On top of that, prediction in a stacked model is longer than in a simple linear model. On the same topic, remember that Netflix never used its 1M$ algorithm because of engineering costs.

I would say that online data science competitions are a good way for a company to know "what is the highest accuracy (or any performance metric) that can be achieved" using the data they collect (at some point in time). Note that this actually is a hard problem which is being solved ! But, in the industry, field knowledge, hardware and business constraints usually discourage the use of "fancy modelling".

Answer 4

Stacking significantly increases complexity and reduces interpretability. The gains are usually relatively small to justify it. So while ensembling is probably widely used (e.g. XGBoost), I think stacking is relatively rare in industry.

Answer 5

In my experience collecting good data and features is much more important.

The clients we worked with usually have a lot of data, and not all of it in format that can be readily exported or easy to work with. The first batch of data is usually not very useful; it is our task to work with the client to figure what data we would need to make the model more useful. This is a very iterative process.

There is a lot of experimentation going on, and we need models that are:

1. Fast to train
2. Fast to predict (Also is often a business requirement)
3. Easy to interpret

Point 3) is especially important, because models that are easy to interpret are easier to communicate to the client and it is easier to catch if we have done something wrong.

Answer 6

Here's something that doesn't come up much on Kaggle: the

• more variables you have in your model, and
• the more complex the relationship between those variables and the output,

the more risk you will face over the lifetime of that model. Time is typically either frozen in Kaggle competitions, or there's a short future time window where test set values come in. In industry, that model might run for years. And all it might take is for one variable to go haywire for your entire model to go to hell, even if it was built flawlessly. I get it, no one wants to watch a contest where competitors carefully balance model complexity against the risk, but out there in a job, your business and quality of life will suffer if something goes wrong with a model you're in charge of. Even extremely smart people aren't immune. Take, for instance, the Google Flu Trends prediction failure. The world changed, and they didn't see it coming.

To O.P.'s question, "In general, in your experience, how important is fancy modelling such as stacking vs simply collecting more data and more features for the data?" Well, I'm officially old, but my answer is that unless you have a really robust modeling infrastructure, it's better to have straightforward models, with a minimal set of variables, where the input-to-output relationship is relatively straightforward. If a variable barely improves your loss metric, leave it out. Remember that it's a job. Get your kicks outside of work on Kaggle contests where there is the "go big or go home" incentive.

One exception would be if the business situation demanded a certain level of model performance, for instance if your company needed to match or beat the performance of a competitor to gain some advantage (probably in marketing). But when there's a linear relationship between the model performance and business gain, the increases in complexity don't typically justify the financial gain (see "Netflix never used its $1 Million Algorithm due to Engineering costs" - apologies to @RUser4512 for citing the same article). In a Kaggle competition however, that additional gain may move you hundreds of ranks as you pass nearby solutions.

Answer 7

A short answer which is a quote I like from Gary Kasparov's book Deep Thinking

A clever process beats superior knowledge and superior technology

I work mainly with time-series financial data, and the process from gathering data, cleaning it, processing it, and then working with the problem owners to figure out what they actually want to do, to then building features and models to try and tackle the problem and finally to retrospectively examine the process to improve for next time.

This whole process is greater than the sum of its parts. I tend to get 'acceptable' generalisation performance with a linear/logistic regression and talking with domain experts to generate features, way better time spent than spending time over-fitting my model to the data I have.