When we read a book, understanding the notations plays a very important role of understanding the contents. Unfortunately, different communities have different notation conventions for the formulation on the model and the optimization problem. Could any one summarize some formulation notations here and provide possible reasons?
I will give an example here: In linear algebra literature, the classic book is Strang's introduction to linear algebra. The most used notation in the book is
$$ A x=b$$
Where $A$ is a coefficient matrix, $x$ is the variables to be solved and $b$ is a vector on right side of the equation. The reason the book choose this notation is the main goal of linear algebra is solving a linear system and figure out what is vector $x$. Given such formulation the OLS optimization problem is
$$ \underset{x}{\text{minimize}}~~ \|A x-b\|^2$$
In statistics or machine learning literate (from book Elements of Statistical Learning) people use different notation to represent the same thing:
$$X \beta= y$$
Where $X$ is the data matrix, $\beta$ is the coefficients or weights to be learned learning, $y$ is the response. The reason people use this is because people in statistics or machine learning community is data driven, so data and response are the most interesting thing to them, where they use $X$ and $y$ to represent.
Now we can see all the possible confusion can be there: $A$ in first equation is as same as $X$ in second equation. And in second equation $X$ is not something need to be solved. Also for the terms: $A$ is the coefficient matrix in linear algebra, but it is data in statistics. $\beta$ is also called "coefficients".
In addition, I mentioned $X \beta=y$ is not the exactly what people widely used in machine learning, people uses a half vectorized version that summarize over all the data points. Such as
$$ \min \sum_i \text{L}(y_i,f(x_i)) $$
I think the reason for this is that it is good when talking about the stochastic gradient descent and other different loss functions. Also, the concise matrix notation disappears for other problems than linear regression.
Matrix notation for logistic regression
Could anyone give more summaries on the notations cross different literature? I hope smart answers to this question can be used as a good reference for people reading books cross different literature.
please do not be limited by my example $A x=b$ and $X \beta=y$. There are many others. Such as
Why there are two different logistic loss formulation / notations?