2
$\begingroup$

I am curious about the differences in the algorithms used to perform a

(multi-) linear regression :

1) Using a Stats package, e.g., Stata, R, Excel

and

2) Through ML in , e.g. , Python.

I mean, the method itself, to my knowledge, is the same:

We find the values of the coefficients that minimize the square of the

differences : $(y_i -y_i^{ hat})^2 $ (Regression coeff - actual data

values)through Calculus by differentiating

and finding the critical values.

EDIT: As I understand it, in ML, we have a loss function, a threshold value.

We split our dataset into test and training data. Then we make an initial

choice of coefficients, evaluate using our test function and if our test

function satisfies the treshold value, if it does, we stop. If not, we

try a different split into test and training data and continue until the

threshold has been met. But in "standard" methods we just use Calculus to

find the coefficients that minimize the difference :

 $\Sigma _{i=1}^k(y= \Beta_1 x_1+....+\Beta_k x_n - \y_i)^2$

Do we get the same outputs using the different methods?

Or are we doing something different in the separate cases?

EDIT2: Apologies, I'm having trouble making Latex render correctly. If

someone can please point out what I am doing wrong.

$\endgroup$

1 Answer 1

2
$\begingroup$

We are not doing anything different in either case, the method remains the same. It is the evaluation of the models which differs between stats and ML.

Machine learning puts prediction as a first class citizen. Thus, model evaluation focuses more on evaluating how the estimator performs out of sample. This leads to cross validation and train/validation/test splits as the main method of model selection and assessment.

Statistics puts inference as the first class citizen. Thus, using as much data as one has available becomes more important than determining if the model can accurately predict new data. In machine learning, a model with high RMSE might not be very valuable, but in statistics high RMSE does not necessarily mean the model is not fruitful.

$\endgroup$
4
  • $\begingroup$ Demetri, is it accurate to say then that we use different loss functions for each case in order to determine how "accurate"(efective?) our model is in terms of prediction or inference? $\endgroup$
    – MSIS
    Commented Jan 3, 2020 at 0:26
  • 1
    $\begingroup$ @MSIS no, not really. Linear regression is a counterexample. Regardless of your goal, prediction vs inference, you still use the same loss function. $\endgroup$ Commented Jan 3, 2020 at 0:48
  • $\begingroup$ So how do we determine if our model is effective towards either aspect, i.e., whether it is effective towards inferences or prediction? $\endgroup$
    – MSIS
    Commented Jan 3, 2020 at 1:00
  • 1
    $\begingroup$ @MSIS In short, you can use cross validation for prediction and you should use knowledge about design and background theory for inference. $\endgroup$ Commented Jan 3, 2020 at 1:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.