What are the classical notations in statistics, linear algebra and machine learning? And what are the connections between these notations?

Question

When we read a book, understanding the notations plays a very important role of understanding the contents. Unfortunately, different communities have different notation conventions for the formulation on the model and the optimization problem. Could any one summarize some formulation notations here and provide possible reasons?

I will give an example here: In linear algebra literature, the classic book is Strang's introduction to linear algebra. The most used notation in the book is

$$ A x=b$$

Where $A$ is a coefficient matrix, $x$ is the variables to be solved and $b$ is a vector on right side of the equation. The reason the book choose this notation is the main goal of linear algebra is solving a linear system and figure out what is vector $x$. Given such formulation the OLS optimization problem is

$$ \underset{x}{\text{minimize}}~~ \|A x-b\|^2$$

In statistics or machine learning literate (from book Elements of Statistical Learning) people use different notation to represent the same thing:

$$X \beta= y$$

Where $X$ is the data matrix, $\beta$ is the coefficients or weights to be learned learning, $y$ is the response. The reason people use this is because people in statistics or machine learning community is data driven, so data and response are the most interesting thing to them, where they use $X$ and $y$ to represent.

Now we can see all the possible confusion can be there: $A$ in first equation is as same as $X$ in second equation. And in second equation $X$ is not something need to be solved. Also for the terms: $A$ is the coefficient matrix in linear algebra, but it is data in statistics. $\beta$ is also called "coefficients".

In addition, I mentioned $X \beta=y$ is not the exactly what people widely used in machine learning, people uses a half vectorized version that summarize over all the data points. Such as

$$ \min \sum_i \text{L}(y_i,f(x_i)) $$

I think the reason for this is that it is good when talking about the stochastic gradient descent and other different loss functions. Also, the concise matrix notation disappears for other problems than linear regression.

Matrix notation for logistic regression

Could anyone give more summaries on the notations cross different literature? I hope smart answers to this question can be used as a good reference for people reading books cross different literature.

please do not be limited by my example $A x=b$ and $X \beta=y$. There are many others. Such as

Why there are two different logistic loss formulation / notations?

Answer 1

Perhaps a related question is, "What are words used in different languages, and what are the connections between these words?"

Notation is in some sense like language:

• Some words have region specific meanings; some words are broadly understood.
• Like powerful nations spread their language, successful fields and influential researchers spread their notation.
• Language evolves over time: language has a mix of historical origins and modern influence.

Your specific question...

• I would disagree with your contention that the two follow "completely different notation." Both $X\boldsymbol{\beta} = \boldsymbol{y}$ and $A\mathbf{x} = \mathbf{b}$ use capital letters to denote matrices. They're not that different.
• Machine learning is highly related to statistics, a large and mature field. Using $X$ to represent the data matrix is almost certainly the most readable, most standard convention to follow. While $A\mathbf{x} = \mathbf{b}$ is standard for solving linear systems, that's not how people doing statistics write the normal equations. You'll find your audience more confused if you try to do that. When in Rome...
• In some sense, the heart of your revised question is, "What are the historical origins of statistics using the letter $x$ to represent data and the letter $\beta$ to represent the unknown variable to solve for?"
  • This is a question for the statistical historians! Briefly searching, I see the influential British statistician and Cambridge academic Udny Yule used $x$ to represent data in his Introduction to the Theory of Statistics (1911). He wrote a regression equation as $x_1 = a + bx_2$, with the least squares objective as minimizing $\sum\left( x_1 - a - bx_2\right)^2$, and with solution $b_{12} = \frac{\sum x_1x_2}{\sum x_2^2}$. It at least goes back to then...
  • The even more influential R.A. Fisher used $y$ for the dependent variable and $x$ for the independent variable in his 1925 book Statistical Methods for Research Workers. (Hat tip to @Nick Cox for providing link with info.)

Good notation is like good language. Avoid field specific jargon whenever possible. Write in the math equivalent of high BBC English, language that is understandable to most anyone that speaks English. One should write, whenever possible, using notation that is clear and that is broadly understood.