Skip to main content
82 votes

Industry vs Kaggle challenges. Is collecting more observations and having access to more variables more important than fancy modelling?

By way of background, I have been doing forecasting store $\times$ SKU time series for retail sales for 12 years now. Tens of thousands of time series across hundreds or thousands of stores. I like ...
Stephan Kolassa's user avatar
45 votes

Industry vs Kaggle challenges. Is collecting more observations and having access to more variables more important than fancy modelling?

I can't speak for the whole of industry, obviously, but I work in industry and have competed on Kaggle so I will share my POV. First, you're right to suspect that Kaggle doesn't exactly match what ...
olooney's user avatar
  • 3,265
42 votes
Accepted

Why should I be Bayesian when my dataset is large?

Being Bayesian is not only about information fed through the prior. But even then: Where the prior is zero, no amount of data will turn that over. Having a full Bayesian posterior distribution to ...
Bernhard's user avatar
  • 8,475
21 votes
Accepted

Is the Kolmogorov-Smirnov-Test too strict if the sample size is large?

With a test you try to find deviations from the Null hypothesis. The larger the sample the better we are at detecting such deviations, even trivially small ones. So if you do test in large samples you ...
Maarten Buis's user avatar
  • 21.2k
21 votes

Industry vs Kaggle challenges. Is collecting more observations and having access to more variables more important than fancy modelling?

From my experience, more data and more features are more important than the fanciest, most stacked, most tuned, model one can come up with. Look at the online advertising competitions that took place....
RUser4512's user avatar
  • 10.4k
20 votes

Why is gradient descent inefficient for large data set?

There are two ways in which gradient descent may be inefficient. Interestingly, they each lead to their own method for fixing up, which are nearly opposite solutions. The two problems are: (1) Too ...
Cliff AB's user avatar
  • 21.3k
18 votes
Accepted

Density estimation for large dataset

There are some tricks you can play with kernel density estimators (KDEs) to improve runtime. For example, you can use a kernel with compact support (e.g. an Epanechnikov or triweight kernel). To ...
user20160's user avatar
  • 32.8k
17 votes
Accepted

Can support vector machine be used in large data?

As you mention, storing the kernel matrix requires memory that scales quadratically with the number of data points. Training time for traditional SVM algorithms also scales superlinearly with the ...
user20160's user avatar
  • 32.8k
16 votes

Industry vs Kaggle challenges. Is collecting more observations and having access to more variables more important than fancy modelling?

Stacking significantly increases complexity and reduces interpretability. The gains are usually relatively small to justify it. So while ensembling is probably widely used (e.g. XGBoost), I think ...
rinspy's user avatar
  • 3,370
16 votes
Accepted

Feature selection for Logistic Regression

My questions is this right approach to do feature selection when data volume is high? Simply, no. Basing feature selection on p values is a bad idea, especially when data are large. First, p-values ...
Demetri Pananos's user avatar
15 votes
Accepted

Training a neural network on chess data

I think you need to consider running it on a GPU. Google Colab is free and Amazon AWS is very cheap. You seem to know what you are doing so you can probably get up and running with PyTorch very ...
Robert Long's user avatar
  • 62.4k
14 votes

Significance test for large sample sizes

The test is doing what it should be doing. You ask it if two quantities are equal, in the case of the original question, if zero is equal to some measure of independence that is zero when the ...
Dave's user avatar
  • 64.2k
13 votes

How should I transform non-negative data including zeros?

Comparing the answer provided in by @RobHyndman to a log-plus-one transformation extended to negative values with the form: $$T(x) = \text{sign}(x) \cdot \log{\left(|x|+1\right)} $$ (As Nick Cox ...
Firebug's user avatar
  • 19.4k
13 votes

Batch Learning w/Random Forest Sklearn

Yes, Batch Learning is certainly possible in scikit-learn. When you first initialize your RandomForestClassifier object you'll want to set the warm_start parameter to True. This means that successive ...
user1993951's user avatar
13 votes
Accepted

In ML, once we remove a feature, can we safely assume that feature will not be important again?

No, you cannot safely assume that. The reason is that conditional independence does not imply independence and vice versa (wiki). Moreover the forward selection style approach you follow suffers from ...
Georg M. Goerg's user avatar
12 votes
Accepted

Things that I am not sure about "LASSO" regression method

For the first question, recall that in centering we replace each value $y_i$ with $y_i - \bar y$, where $\bar y$ is the mean of the $y$ vector. Then $$ \sum_i (y_i - \bar y) = \sum_i y_i - n \bar y =...
Matthew Drury's user avatar
12 votes

How to run linear regression in a parallel/distributed way for big data setting?

Short Answer: Yes, running linear regression in parallel has been done. For example, Xiangrui Meng et al. (2016) for Machine Learning in Apache Spark. The way it works is using stochastic gradient ...
Haitao Du's user avatar
  • 37.1k
12 votes
Accepted

What do high dimensional cauchy distributions look like?

The main challenge in this question lies in interpreting the sense of "accumulate around some manifold." The difficulty is that no such thing can happen, because as the vector length $d$ ...
whuber's user avatar
  • 327k
11 votes
Accepted

How can I quickly detect cheating variables in large data?

This is sometimes referred to as "Data Leakage." There's a nice paper on this here: Leakage in Data Mining: Formulation, Detection, and Avoidance The above paper has plenty of amusing (and ...
Alex R.'s user avatar
  • 14k
11 votes

Why is gradient descent inefficient for large data set?

First let me suggest an improvement to your notation. In particular, let's denote the loss function by $L(w)$ rather than $f(x)$. Using the letter $L$ is simply a personal preference of mine since it ...
tddevlin's user avatar
  • 3,387
11 votes
Accepted

Is exploratory data analysis (EDA) actually needed / useful

I come from a traditional biostatistics/epidemiology background, and EDA are definitely useful, although it doesn't mean doing histograms/correlation plots just for the sake of it. With the ...
Tim Mak's user avatar
  • 1,182
10 votes
Accepted

When can I stop looking for a better model?

You might be interested in a formal science domain called "computational mechanics." In an article by James Crutchfield and David Feldman, they lay out the program of computational mechanics—as far as ...
Alexis's user avatar
  • 30.2k
10 votes

In ML, once we remove a feature, can we safely assume that feature will not be important again?

You seem to be assuming that the models work in additive fashion, so adding a feature to the model just "adds" some stuff related to this feature alone and does not influence the rest of the model, ...
Tim's user avatar
  • 139k
10 votes
Accepted

Why does the condition number of the covariance matrix explode as number of variables increases?

Explaining this in the comments was a little limiting, apologies: Assuming centered data matrix $X$, then your covariance matrix $M = X^T X$. This will have high condition number if the range of ...
proof_by_accident's user avatar
10 votes
Accepted

Bayesian stats and multiple tests

Yes, looking at multiple questions and deciding "reject one or more null hypotheses" based on a Bayesian analysis (e.g. based on some cut-off for a Bayes factor, or if posterior credible ...
Björn's user avatar
  • 33.2k
9 votes

Are large data sets inappropriate for hypothesis testing?

"Does it mean that hypothesis testing is worthless for large data sets?" No, it doesn't mean that. The general message is that decisions made after conducting a hypothesis test should always take ...
Zen's user avatar
  • 24.4k
9 votes

How to run linear regression in a parallel/distributed way for big data setting?

As @hxd1011 mentioned, one approach is to formulate linear regression as an optimization problem, then solve it using an iterative algorithm (e.g. stochastic gradient descent). This approach can be ...
user20160's user avatar
  • 32.8k
9 votes

Industry vs Kaggle challenges. Is collecting more observations and having access to more variables more important than fancy modelling?

In my experience collecting good data and features is much more important. The clients we worked with usually have a lot of data, and not all of it in format that can be readily exported or easy to ...
Akavall's user avatar
  • 2,671
9 votes

Advice on running random forests on a large dataset

Some hints: 500k rows with 100 columns do not impose problems to load and prepare, even on a normal laptop. No need for big data tools like spark. Spark is good in situations with hundreds of ...
Michael M's user avatar
  • 11.8k
9 votes

Why should I be Bayesian when my dataset is large?

I'd like to echo some of the points in the other answer with slightly different emphasis. To me the most important issue is that the Bayesian view of uncertainty/probability/randomness is the one that ...
daniel.s's user avatar
  • 280

Only top scored, non community-wiki answers of a minimum length are eligible