4
$\begingroup$

Many libraries that scale linear and logistic regression assume a tall-skinny design matrix (many samples, few features), but I don't understand why you would need billions of samples if your data has 250 features.

In what scenarios would more data help? It seems like, instead of using more computational resources, you could simply sub-sample the data and achieve comparable accuracy in scenarios where the feature count is relatively small.

When feature count is high, say 100k, would having samples on the order of billions help? How do we decide how many samples are needed for a given modeling problem?

Perhaps more data helps with imbalanced data? e.g. when performing binary logistic regression for outlier detection, where the occurrence of positive samples is very small.

Any help here would be greatly appreciated.

$\endgroup$
4
  • $\begingroup$ Maybe it really matters if you can predict a probability of $0.5001$ instead of $0.5000$. To get that kind of resolution, you might need a lot of samples. $\endgroup$
    – Dave
    Jul 30 at 19:24
  • $\begingroup$ do you have a specific paper/model at mind? They usually include motivation for their research. Also, scale means different things for different people, you have papers that deal with situations with billions of samples for 250 features, but also for 100k features. $\endgroup$
    – rep_ho
    Jul 30 at 20:08
  • 2
    $\begingroup$ "250 features" really means only 250 features given to you. As soon as you consider nonlinear relations and interactions in your modeling, the number of potential features to include explodes. That design matrix might not look so skinny after all ;-). $\endgroup$
    – whuber
    Jul 30 at 20:51
  • $\begingroup$ Can you give examples of libraries, the classic example I know of is vowpal Wabbit that scales for large number of features ( basically sparse features from one hot encoding) and large number of samples ( to handle the disjoint sets of one hot encoding) eg estimating the click through rate of Amazons product catalog $\endgroup$
    – seanv507
    Jul 30 at 21:38
4
$\begingroup$

Facebook's revenue in 2020 was 86 billion dollars, mostly by selling ads. So if your large scale logistic regression model can improve the click-through rate by 0.01%, you would make facebook 8.6 million dollars per year (86000000000*0.0001), which is enough money to justify spending a couple of millions on some grad students or engineers to work on that.

I don't know if there is a use case for fitting 250 PARAMETERS on billions of samples. But fitting tens of thousands of parameters is quite common. A paper might showcase a method on the 250 parameter toy problem, but my guess would be the real motivation is to fit bigger models. Also, these papers usually don't just fit your vanilla logistic regression, but some kind of regularized generalized linear model and a lot of things can be expressed as regularized logistic regression or GLM.

Even without being Facebook, you can find a problem that could be expressed as a model with billions of samples and hundreds of thousands of parameters. A lot of things can explode really quickly if you allow them to. For example, on a map, each point in a grid is a separate observation that, of course, needs to be modeled to take spatial relationships into account. Nevertheless, all of a sudden, you might have millions of "data points." You might have a time series, where of course, each measurement is a separate data point, and if you measure every millisecond, or if you measure for a long time, all of a sudden, you have millions or billions of data points. If you measure how something spatial changes across time, possibly with respect to some variables, e.g., fMRI signal in a brain, EEG signal on a scalp, smog in a city, global temperature, algae on a lake... you can easily get a problem of a scale you were talking about.

These problems might not always be looked at as large-scale logistic regression problems, but if there was a technology to efficiently fit those models, use cases would come by themselves.

tl;dr: time series, maps, and images. Could you just downsample it? Yes, but you wouldn't need to.

$\endgroup$
2
$\begingroup$

Your general premise is correct: you don’t need more data than you need. But also, as noticed in the other answer and comments, it may be more tricky than you’re assuming. First of all you need large enough, high-quality sample of data. Using more data than is needed for the desired precision would be wasteful. On another hand, logistic regression is a pretty fast and cheap model to train, at least as compared with many other machine learning models, so you wouldn’t usually care that much. You want the libraries to be scalable for cases where it’s needed, but this doesn’t mean that you always need to push as much data as you can to the algorithm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.