Return to Answer

I can't speak for the whole of industry, obviously, but I work in industry and have competed Kaggle so I will share my POV.

First, you're right to suspect that Kaggle doesn't exactly match what people are doing in industry. It's a game, and subject to gamesmanship, with lots of crazy restrictions. For example, in the currently running Santander competition:

1. The feature names were artificially hashed to hide their meaning
2. The "training" set was artificially limited to have fewer rows than columns specifically so that feature selection, robustness, and regularization technique would be indispensable to success.
3. The so-called "test" set has a markedly different distribution than the training set and the two are clearly not random samples from the same population.

If someone gave me a data set like this at work, I would immediately offer to work with them on feature engineering so we could get features that were more useful. I would suggest we use domain knowledge to decide on likely interaction terms, thresholds, categorical variable coding strategies, etc. Approaching the problem in that way would clearly be more productive than trying to extract meaning from an exhaust file produced by a database engineer with no training in ML.

Furthermore, if you learn, say, that a particular numeric column is not numeric at all but rather a ZIP code, well, you can go and get data from 3rd-party data sources such as the US Census to augment your data. Or if you have a date, maybe you'll include the S&P 500 closing price for that day. Such external augmentation strategies require detailed knowledge of the specific data set and significant domain knowledge but usually have the much larger payoffs than pure algorithmic improvements.

So, the first big difference between industry and Kaggle is that in industry, features (in the sense of input data) are negotiable.

A second class of differences is performance. Often, models will be deployed to production in one of two ways: 1) model predictions will be pre-computed for every row in a very large database table, or 2) an application or website will pass the model a single row of data and need a prediction returned in real-time. Both use cases require good performance. For these reasons, you don't often see models that can be slow to predict or use a huge amount of memory like K-Nearest-Neighbors or Extra Random Forests. A logistic regression or neural network, in contrast, can score a batch of records with a few matrix multiplications, and matrix multiplication can be highly optimized with the right libraries. Even though I could get maybe +0.001 AUC if I stacked on yet another non-parametric model, I wouldn't because prediction throughput and latency would drop too much.

There's a practical dimension to this as well - stacking four different state-of-the-art 3rd-party libraries, say LightGBM, xgboost, catboost, and Tensorflow (on GPUs, of course) might get you that .01 reduction in MSE that wins Kaggle competitions, but it's four different libraries to install, deploy, and debug if something goes wrong. It's great if you can get all that stuff working on your laptop, but getting it running inside a Docker container running on AWS is a completely different story. Most companies don't want to front a small devops team just to deal with these kinds of deployment issues.

That said, stacking in itself isn't necessarily a huge deal. In fact, stacking a couple different models that all perform equally well but have very different decision boundaries is a great way to get a small bump in AUC and a big bump in robustness. Just don't go throwing so many kitchen sinks into your heterogeneous ensemble that you start to have deployment issues.

I can't speak for the whole of industry, obviously, but I work in industry and have competed on Kaggle so I will share my POV.

First, you're right to suspect that Kaggle doesn't exactly match what people are doing in industry. It's a game, and subject to gamesmanship, with lots of crazy restrictions. For example, in the currently running Santander competition:

1. The feature names were artificially hashed to hide their meaning
2. The "training" set was artificially limited to have fewer rows than columns specifically so that feature selection, robustness, and regularization technique would be indispensable to success.
3. The so-called "test" set has a markedly different distribution than the training set and the two are clearly not random samples from the same population.

If someone gave me a data set like this at work, I would immediately offer to work with them on feature engineering so we could get features that were more useful. I would suggest we use domain knowledge to decide on likely interaction terms, thresholds, categorical variable coding strategies, etc. Approaching the problem in that way would clearly be more productive than trying to extract meaning from an exhaust file produced by a database engineer with no training in ML.

Furthermore, if you learn, say, that a particular numeric column is not numeric at all but rather a ZIP code, well, you can go and get data from 3rd-party data sources such as the US Census to augment your data. Or if you have a date, maybe you'll include the S&P 500 closing price for that day. Such external augmentation strategies require detailed knowledge of the specific data set and significant domain knowledge but usually have the much larger payoffs than pure algorithmic improvements.

So, the first big difference between industry and Kaggle is that in industry, features (in the sense of input data) are negotiable.

A second class of differences is performance. Often, models will be deployed to production in one of two ways: 1) model predictions will be pre-computed for every row in a very large database table, or 2) an application or website will pass the model a single row of data and need a prediction returned in real-time. Both use cases require good performance. For these reasons, you don't often see models that can be slow to predict or use a huge amount of memory like K-Nearest-Neighbors or Extra Random Forests. A logistic regression or neural network, in contrast, can score a batch of records with a few matrix multiplications, and matrix multiplication can be highly optimized with the right libraries. Even though I could get maybe +0.001 AUC if I stacked on yet another non-parametric model, I wouldn't because prediction throughput and latency would drop too much.

There's a reliability dimension to this as well - stacking four different state-of-the-art 3rd-party libraries, say LightGBM, xgboost, catboost, and Tensorflow (on GPUs, of course) might get you that .01 reduction in MSE that wins Kaggle competitions, but it's four different libraries to install, deploy, and debug if something goes wrong. It's great if you can get all that stuff working on your laptop, but getting it running inside a Docker container running on AWS is a completely different story. Most companies don't want to front a small devops team just to deal with these kinds of deployment issues.

That said, stacking in itself isn't necessarily a huge deal. In fact, stacking a couple different models that all perform equally well but have very different decision boundaries is a great way to get a small bump in AUC and a big bump in robustness. Just don't go throwing so many kitchen sinks into your heterogeneous ensemble that you start to have deployment issues.

I can't speak for the whole of industry, obviously, but I work in industry and have competed Kaggle so I will share my POV.

First, you're right to suspect that Kaggle doesn't exactly match what people are doing in industry. It's a game, and subject to gamesmanship, with lots of crazy restrictions. For example, in the currently running Santander competition:

1. The feature names were artificially hashed to hide their meaning
2. The "training" set was artificially limited to have fewer rows than columns specifically so that feature selection, robustness, and regularization techniques.
3. The so-called "test" set has a markedly different distribution than the training set and the two are clearly not random samples from the same population.

If someone gave me a data set like this at work, I would immediately offer to work with them on feature engineering so we could get features that were more useful. I would suggest we use domain knowledge to decide on likely interaction terms, thresholds, categorical variable coding strategies, etc. Approaching the problem in that way would clearly be more productive than trying to extract meaning from an exhaust file produced by a database engineer with no training in ML.

Furthermore, if you learn, say, that a particular numeric column is not numeric at all but rather a ZIP code, well, you can go and get data from 3rd-party data sources such as the US Census to augment your data. Or if you have a date, maybe you'll include the S&P 500 closing price for that day. Such external augmentation strategies require detailed knowledge of the specific data set and significant domain knowledge but usually have the much larger payoffs than pure algorithmic improvements.

So, the first big difference between industry and Kaggle is that in industry, features (in the sense of input data) are negotiable.

A second class of differences is performance. Often, models will be deployed to production in one of two ways: 1) model predictions will be pre-computed for every row in a very large database table, or 2) an application or website will pass the model a single row of data and need a prediction returned in real-time. Both use cases require good performance. For these reasons, you don't often see models that can be slow to predict or use a huge amount of memory like K-Nearest-Neighbors or Extra Random Forests. A logistic regression or neural network, in contrast, can score a batch of records with a few matrix multiplications, and matrix multiplication can be highly optimized with the right libraries. Even though I could get maybe +0.001 AUC if I stacked on yet another non-parametric model, I wouldn't because prediction throughput and latency would drop too much.

There's a practical dimension to this as well - stacking four different state-of-the-art 3rd-party libraries, say LightGBM, xgboost, catboost, and Tensorflow (on GPUs, of course) might get you that .01 reduction in MSE that wins Kaggle competitions, but it's four different libraries to install, deploy, and debug if something goes wrong. It's great if you can get all that stuff working on your laptop, but getting it running inside a Docker container running on AWS is a completely different story. Most companies don't want to front a small devops team just to deal with these kinds of deployment issues.

That said, stacking in itself isn't necessarily a huge deal. In fact, stacking a couple different models that all perform equally well but have very different decision boundaries is a great way to get a small bump in AUC and a big bump in robustness. Just don't go throwing so many kitchen sinks into your heterogeneous ensemble that you start to have deployment issues.

I can't speak for the whole of industry, obviously, but I work in industry and have competed Kaggle so I will share my POV.

First, you're right to suspect that Kaggle doesn't exactly match what people are doing in industry. It's a game, and subject to gamesmanship, with lots of crazy restrictions. For example, in the currently running Santander competition:

1. The feature names were artificially hashed to hide their meaning
2. The "training" set was artificially limited to have fewer rows than columns specifically so that feature selection, robustness, and regularization technique would be indispensable to success.
3. The so-called "test" set has a markedly different distribution than the training set and the two are clearly not random samples from the same population.

If someone gave me a data set like this at work, I would immediately offer to work with them on feature engineering so we could get features that were more useful. I would suggest we use domain knowledge to decide on likely interaction terms, thresholds, categorical variable coding strategies, etc. Approaching the problem in that way would clearly be more productive than trying to extract meaning from an exhaust file produced by a database engineer with no training in ML.

Furthermore, if you learn, say, that a particular numeric column is not numeric at all but rather a ZIP code, well, you can go and get data from 3rd-party data sources such as the US Census to augment your data. Or if you have a date, maybe you'll include the S&P 500 closing price for that day. Such external augmentation strategies require detailed knowledge of the specific data set and significant domain knowledge but usually have the much larger payoffs than pure algorithmic improvements.

So, the first big difference between industry and Kaggle is that in industry, features (in the sense of input data) are negotiable.

A second class of differences is performance. Often, models will be deployed to production in one of two ways: 1) model predictions will be pre-computed for every row in a very large database table, or 2) an application or website will pass the model a single row of data and need a prediction returned in real-time. Both use cases require good performance. For these reasons, you don't often see models that can be slow to predict or use a huge amount of memory like K-Nearest-Neighbors or Extra Random Forests. A logistic regression or neural network, in contrast, can score a batch of records with a few matrix multiplications, and matrix multiplication can be highly optimized with the right libraries. Even though I could get maybe +0.001 AUC if I stacked on yet another non-parametric model, I wouldn't because prediction throughput and latency would drop too much.

There's a practical dimension to this as well - stacking four different state-of-the-art 3rd-party libraries, say LightGBM, xgboost, catboost, and Tensorflow (on GPUs, of course) might get you that .01 reduction in MSE that wins Kaggle competitions, but it's four different libraries to install, deploy, and debug if something goes wrong. It's great if you can get all that stuff working on your laptop, but getting it running inside a Docker container running on AWS is a completely different story. Most companies don't want to front a small devops team just to deal with these kinds of deployment issues.

That said, stacking in itself isn't necessarily a huge deal. In fact, stacking a couple different models that all perform equally well but have very different decision boundaries is a great way to get a small bump in AUC and a big bump in robustness. Just don't go throwing so many kitchen sinks into your heterogeneous ensemble that you start to have deployment issues.

I can't speak for the whole of industry, obviously, but I work in industry and have competed Kaggle so I will share my POV.

First, you're right to suspect that Kaggle doesn't exactly match what people are doing in industry. It's a game, and subject to gamesmanship, with lots of crazy restrictions. For example, in the currently running Santander competition:

1. The feature names were artificially hashed to hide their meaning
2. The "training" set was artificially limited to have fewer rows than columns specifically so that feature selection, robustness, and regularization techniques.
3. The so-called "test" set has a markedly different distribution than the training set and the two are clearly not random samples from the same population.

If someone gave me a data set like this at work, I would immediately offer to work with them on feature engineering so we could get features that didn't suck, and would use domain knowledge to decide on likely interaction terms, thresholds, categorical variable coding strategies, etc. Approaching the problem in that way would clearly be more productive than trying to extract meaning from an exhaust file produced by a database engineer not training in ML.

Furthermore, if you learn, say, that a particular numeric column is not numeric at all but rather a ZIP code, well, you can go and get data from 3rd-party data sources such as the US Census to augment your data. Or if you have a date, maybe you'll include the S&P 500 closing price for that day. Such external augmentation strategies usually have the much larger payoffs than pure algorithmic improvements.

So, the first big difference between industry and Kaggle is that in industry, features (in the sense of input data) are negotiable.

A second class of differences is performance. Often, models that will be deployed to production in one of two ways: either model predictions will be pre-computed for every row in a very large database table, or an application or website will pass the model a single row of data and need a prediction returned in real-time. For these reasons, you don't often see models that can be slow to predict or use a huge amount of memory like K-Nearest-Neighbors or Extra Random Forests. A logistic regression or neural network, in contrast, can score a batch of records with a few matrix multiplications, and matrix multiplication can be highly optimized with the right libraries. Even though I could get maybe +0.001 AUC if I added yet another non-parametric, I wouldn't because prediction throughput and latency would drop too much.

There's a practical dimension to this as well - stacking four different state-of-the-art 3rd-party libraries, say LightGBM, xgboost, catboost, and Tensorflow (on GPUs, of course) might get you that .01 reduction in MSE that wins Kaggle competitions, but it's four different libraries to install, deploy, and debug if something goes wrong. It's great if you can get all that stuff working on your laptop, but getting it running inside a Docker container running on AWS is a completely different story. Most companies don't want to front a small devops team just to deal with these kinds of deployment issues.

That said, stacking in itself isn't necessarily a huge deal. In fact, stacking a couple different models that all perform equally well but have very different decision boundaries is a great way to get a small bump in AUC and a big bump in robustness. Just don't go throwing so many kitchen sinks into your heterogeneous ensemble that you start to have deployment issues.

I can't speak for the whole of industry, obviously, but I work in industry and have competed Kaggle so I will share my POV.

First, you're right to suspect that Kaggle doesn't exactly match what people are doing in industry. It's a game, and subject to gamesmanship, with lots of crazy restrictions. For example, in the currently running Santander competition:

1. The feature names were artificially hashed to hide their meaning
2. The "training" set was artificially limited to have fewer rows than columns specifically so that feature selection, robustness, and regularization techniques.
3. The so-called "test" set has a markedly different distribution than the training set and the two are clearly not random samples from the same population.

If someone gave me a data set like this at work, I would immediately offer to work with them on feature engineering so we could get features that were more useful. I would suggest we use domain knowledge to decide on likely interaction terms, thresholds, categorical variable coding strategies, etc. Approaching the problem in that way would clearly be more productive than trying to extract meaning from an exhaust file produced by a database engineer with no training in ML.

Furthermore, if you learn, say, that a particular numeric column is not numeric at all but rather a ZIP code, well, you can go and get data from 3rd-party data sources such as the US Census to augment your data. Or if you have a date, maybe you'll include the S&P 500 closing price for that day. Such external augmentation strategies require detailed knowledge of the specific data set and significant domain knowledge but usually have the much larger payoffs than pure algorithmic improvements.

So, the first big difference between industry and Kaggle is that in industry, features (in the sense of input data) are negotiable.

A second class of differences is performance. Often, models will be deployed to production in one of two ways: 1) model predictions will be pre-computed for every row in a very large database table, or 2) an application or website will pass the model a single row of data and need a prediction returned in real-time. Both use cases require good performance. For these reasons, you don't often see models that can be slow to predict or use a huge amount of memory like K-Nearest-Neighbors or Extra Random Forests. A logistic regression or neural network, in contrast, can score a batch of records with a few matrix multiplications, and matrix multiplication can be highly optimized with the right libraries. Even though I could get maybe +0.001 AUC if I stacked on yet another non-parametric model, I wouldn't because prediction throughput and latency would drop too much.

There's a practical dimension to this as well - stacking four different state-of-the-art 3rd-party libraries, say LightGBM, xgboost, catboost, and Tensorflow (on GPUs, of course) might get you that .01 reduction in MSE that wins Kaggle competitions, but it's four different libraries to install, deploy, and debug if something goes wrong. It's great if you can get all that stuff working on your laptop, but getting it running inside a Docker container running on AWS is a completely different story. Most companies don't want to front a small devops team just to deal with these kinds of deployment issues.

That said, stacking in itself isn't necessarily a huge deal. In fact, stacking a couple different models that all perform equally well but have very different decision boundaries is a great way to get a small bump in AUC and a big bump in robustness. Just don't go throwing so many kitchen sinks into your heterogeneous ensemble that you start to have deployment issues.