2
$\begingroup$

Is common to say that Machine Learning techniques represent are purely data driven methods, and them are effective only if we have a large amount of data. I focused here on supervised/predictive learning. If we intend “large” as large number of eligible predictors I agree. However some people says that also a large number of observations needed. However I’m dubious about that because one key result behind the opportunity of predictive learning is the bias-variance tradeoff (see here can help: Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict") Bias/variance tradeoff tutorial Question about bias-variance tradeoff Endogeneity in forecasting ). Then, is possible to show that if the amount of observations go to infinity the tradeoff disappear and only bias become relevant. Now, In order to address the bias the theoretical knowledge about the phenomena under investigation are much more important than the computational aspects. Therefore it seems me that more observation we have more important the theory become. If what I said is correct the opposite of the underscored phrase is true: few data are good situation in predictive learning, even if less observations we have and more simple the (predictive) model should be.

My mistakes? Maybe the truth is in the middle?

Improve this question
$\endgroup$
9
  • 3
    $\begingroup$ When people say a large amount of data, they mean a large number of observations. A large amount of variables would be expressed as high-dimensional data. For machine learning, you need a large amount of data, not high-dimensional data. If I find the time, I will post a more complete answer; for now this is just a comment. $\endgroup$
    – Richard Hardy
    Commented Oct 9, 2020 at 13:28
  • $\begingroup$ I’m not an expert in machine learning and I can make big mistakes. However, staying at the nomenclature that you suggest, it sound me strange that “For machine learning, you need a large amount of data, not high-dimensional data”. Now, if you intend that in ML $p>N$ is not needed … that’s ok. But if you intend that small value for $p$ are usual in ML … it seems me wrong. At least in linear regression examples, the tool in which i’m more confident, my main argument is that in ML something like “automated predictors selection rule” is the core. $\endgroup$
    – markowitz
    Commented Oct 9, 2020 at 19:13
  • 1
    $\begingroup$ At the other side, in classical linear regression framework, tools as stepwise selection are considered very dangerous. Infact prof Studenmund call it as “right tool for bad econometrics”. He suggest theory driven selection. This suggestion work much better if $p$ is small … independent variables come from theory at first round … endogeneity is the core. In linear regression the distinction between classical stat and ML are addressed here (stats.stackexchange.com/questions/268755/…) $\endgroup$
    – markowitz
    Commented Oct 9, 2020 at 19:14
  • $\begingroup$ reply of david25272 is what I appreciate more, and in it the many predictors are typical for ML. Moreover, if only large $N$ is what we need for ML … it seems that it boil down in usual asymptotic theory. No? Surely I will be able to understand more when/if you find time for more exhaustive answer. $\endgroup$
    – markowitz
    Commented Oct 9, 2020 at 19:14
  • $\begingroup$ Hi Richard, even your suggestions here would be appreciated: stats.stackexchange.com/questions/497271/… $\endgroup$
    – markowitz
    Commented Nov 22, 2020 at 13:50

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.