I learning some machine learning course, and I would like to know in which case we use linea algebra and Matrix Algebra?

Thank you

Kind regards

  • 8
    $\begingroup$ Maybe the list will be shorter if you ask which machine learning algorithms don't use linear algebra and matrix algebra. $\endgroup$ – Mark L. Stone Aug 22 '16 at 15:58
  • $\begingroup$ Ok so, I change my question like you said :) $\endgroup$ – Poisson Aug 22 '16 at 16:05
  • $\begingroup$ Have any suggestion for a title ? $\endgroup$ – Poisson Aug 22 '16 at 17:03

I agree with Mark L. Stone's comment: too many examples, linear algebra is the foundation of statistics and machine learning, because your data is organized in columns and rows, and most widely used operations are multiplication and addition.

I will use linear regression as an example.

Suppose you are doing multiple linear regression, where you want to build a model for data $x^{(i)}$ (the $i$th data in your data set, and assume your data as $m$ features.)

$$ f(x^{(i)})=\beta_0+\beta_1x_1^{(i)}+\beta_2x_2^{(i)}+\beta_3x_3^{(i)}+\cdots+\beta_mx_m^{(i)} $$

And you want to minimize the squared error

$$ \sum_i (y^{(i)}-f(x^{(i)}))^2 $$

Such problem can be written in a very concise notation

$$ \min \|X\beta-y\|^2 $$

Where $X$ is data matrix, every row is a data instance, and every column is a "feature". $\beta$ is a vector and $y$ is the response vector. $\|\cdot\|$ is $L_2$ norm of a vector.

Here is where the matrix algebra comes in: take the derivative respect to $\beta$ of $\|X\beta-y\|^2$, you get $2X^T(X^T-y)$. You can set it to $0$ and solve it, which is the "normal equation" for linear regression, where

$$ \beta=(X^TX)^{-1}X^Ty $$

Note that, using matrix inverse is not a good solution from numerical analysis point of view. In real world, such as in R, QR decomposition is used. And that is more matrix algebra.

Check this post, you will see matrix operations are everywhere

What algorithm is used in linear regression?

| cite | improve this answer | |
  • $\begingroup$ thank you very much, to make it simple here : stats.stackexchange.com/questions/230974/diagnosis-analytics my collegue suggest to me to use lean algebra to resolve this problem. Are you agree with that? Kind regards $\endgroup$ – Poisson Aug 22 '16 at 16:52
  • $\begingroup$ it depends how far you want to go. if you want to build some "model" that somehow "works", and you do not want to totally understand the model. you can skip the math. Otherwise, it is good to know the math. $\endgroup$ – Haitao Du Aug 22 '16 at 16:56
  • $\begingroup$ Yes, but Are you Ok with this idea to detect the equipement if you have read the post please? Thank you $\endgroup$ – Poisson Aug 22 '16 at 17:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.