7
$\begingroup$

I had the understanding that the major difference between machine learning and statistical model is, the later "assumes" certain type of distribution of data & based on that different model paradigm as well as statistical results we obtain (e.g. p-values, F-statistics, t-stat, etc.). But in case of machine learning, we don't bother about distribution of data and more interested in prediction.

When I was going through Mllib doc, I found for linear regression we are specifying a distribution. But Mllib is a machine learning package. So, I've the following questions:

1) Is my understanding between ML & statistical method is wrong?

2) Is spark is using statistical modeling for linear regression and GLMs?

Thanks!

Note: There are lot of wonderful post regarding the difference between machine learning and statistical method. But this more related to spark MLLIB.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$
  1. Unfortunately, the dichotomy you describe is invalid. ML models (almost always) define a response distribution. For example, the extremely popular gradient boosting machine library XGBoost defines particular learning objectives (e.g. linear, logistic, Poisson, Cox, etc.).
  2. The implementation of linear regression and GLMs in Spark's MLlib is definitely based on standard Statistical theory for linear models. For example, quoting directly from pyspark/mllib/regression.py's LinearRegressionWithSGD comments: Train a linear regression model using Stochastic Gradient Descent (SGD). This solves the least squares regression formulation f(weights) = 1/(2n) ||A weights - y||^2 which is the mean squared error. i.e. this is a standard linear regression algorithm for Gaussian response. The implementation of the particular algorithm might be optimised such that it works for very large datasests (see for example this excellent thread on "Why use gradient descent for linear regression, when a closed-form math solution is available?") but the theory behind an algorithm is exactly the same.
Improve this answer
$\endgroup$
3
  • $\begingroup$ I agree with user11852. I would like to add that a response distribution is also needed in a ML approach because depending on that you choose a loss/cost function. $\endgroup$
    – fabiob
    Commented May 29, 2018 at 7:22
  • $\begingroup$ Just curious. If ML methods also follows distribution, why they don't provide p-values, t-statiscs, etc? Apart from the fct that ML practioners are more interested in prediction rather than detail statistical significance of the model. Thanks! $\endgroup$
    – Beta
    Commented May 29, 2018 at 16:16
  • $\begingroup$ @Beta: Pretty much you answered your side-question. Practitioners usually are focused on prediction and do not need/want to isolate the effect of a special variable like a treatment/intervention. Also in most cases, the sample size is considered huge so we (probably misguidedly) expect everything to be of (some) real significance. $\endgroup$
    – usεr11852
    Commented May 29, 2018 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.