26
$\begingroup$

When we read a book, understanding the notations plays a very important role of understanding the contents. Unfortunately, different communities have different notation conventions for the formulation on the model and the optimization problem. Could any one summarize some formulation notations here and provide possible reasons?

I will give an example here: In linear algebra literature, the classic book is Strang's introduction to linear algebra. The most used notation in the book is

$$ A x=b$$

Where $A$ is a coefficient matrix, $x$ is the variables to be solved and $b$ is a vector on right side of the equation. The reason the book choose this notation is the main goal of linear algebra is solving a linear system and figure out what is vector $x$. Given such formulation the OLS optimization problem is

$$ \underset{x}{\text{minimize}}~~ \|A x-b\|^2$$

In statistics or machine learning literate (from book Elements of Statistical Learning) people use different notation to represent the same thing:

$$X \beta= y$$

Where $X$ is the data matrix, $\beta$ is the coefficients or weights to be learned learning, $y$ is the response. The reason people use this is because people in statistics or machine learning community is data driven, so data and response are the most interesting thing to them, where they use $X$ and $y$ to represent.

Now we can see all the possible confusion can be there: $A$ in first equation is as same as $X$ in second equation. And in second equation $X$ is not something need to be solved. Also for the terms: $A$ is the coefficient matrix in linear algebra, but it is data in statistics. $\beta$ is also called "coefficients".

In addition, I mentioned $X \beta=y$ is not the exactly what people widely used in machine learning, people uses a half vectorized version that summarize over all the data points. Such as

$$ \min \sum_i \text{L}(y_i,f(x_i)) $$

I think the reason for this is that it is good when talking about the stochastic gradient descent and other different loss functions. Also, the concise matrix notation disappears for other problems than linear regression.

Matrix notation for logistic regression

Could anyone give more summaries on the notations cross different literature? I hope smart answers to this question can be used as a good reference for people reading books cross different literature.

please do not be limited by my example $A x=b$ and $X \beta=y$. There are many others. Such as

Why there are two different logistic loss formulation / notations?

Improve this question
$\endgroup$
10
  • 5
    $\begingroup$ Notation doesn't really exist as some sort of externally verifiable truth. It's a language, so it's inherently contextual and up for redefinition. If I write x*b and say that means matrix x dot product vector b, it just is, bolded or not. $\endgroup$
    – Sycorax ♦
    Jun 4, 2016 at 15:49
  • 3
    $\begingroup$ I'd say that $Ax = b$ and $X \beta = y$ have equivalent notation. Just the names of the variables have changed. In general, you won't find consistent naming of the variables from paper to paper, even within a field. $\endgroup$
    – user20160
    Jun 4, 2016 at 17:16
  • 6
    $\begingroup$ At present, this has 10 upvotes, 150 views; it seems to be a valued & useful thread. Moreover, it has an upvoted answer; so I don't think it's too broad to be answered. $\endgroup$
    – gung - Reinstate Monica
    Jun 4, 2016 at 23:47
  • 3
    $\begingroup$ I agree with @gung, the community clearly has some interest in this question. I nominated for reopening. $\endgroup$
    – Matthew Drury
    Jun 5, 2016 at 23:06
  • 1
    $\begingroup$ I think it's too broad for a regular q. - but as it's already CW & somewhat popular, I've added my vote to re-open to the four that were there. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Jun 6, 2016 at 8:37

1 Answer 1

Reset to default
18
$\begingroup$

Perhaps a related question is, "What are words used in different languages, and what are the connections between these words?"

Notation is in some sense like language:

  • Some words have region specific meanings; some words are broadly understood.
  • Like powerful nations spread their language, successful fields and influential researchers spread their notation.
  • Language evolves over time: language has a mix of historical origins and modern influence.

Your specific question...

  • I would disagree with your contention that the two follow "completely different notation." Both $X\boldsymbol{\beta} = \boldsymbol{y}$ and $A\mathbf{x} = \mathbf{b}$ use capital letters to denote matrices. They're not that different.
  • Machine learning is highly related to statistics, a large and mature field. Using $X$ to represent the data matrix is almost certainly the most readable, most standard convention to follow. While $A\mathbf{x} = \mathbf{b}$ is standard for solving linear systems, that's not how people doing statistics write the normal equations. You'll find your audience more confused if you try to do that. When in Rome...
  • In some sense, the heart of your revised question is, "What are the historical origins of statistics using the letter $x$ to represent data and the letter $\beta$ to represent the unknown variable to solve for?"
    • This is a question for the statistical historians! Briefly searching, I see the influential British statistician and Cambridge academic Udny Yule used $x$ to represent data in his Introduction to the Theory of Statistics (1911). He wrote a regression equation as $x_1 = a + bx_2$, with the least squares objective as minimizing $\sum\left( x_1 - a - bx_2\right)^2$, and with solution $b_{12} = \frac{\sum x_1x_2}{\sum x_2^2}$. It at least goes back to then...
    • The even more influential R.A. Fisher used $y$ for the dependent variable and $x$ for the independent variable in his 1925 book Statistical Methods for Research Workers. (Hat tip to @Nick Cox for providing link with info.)

Good notation is like good language. Avoid field specific jargon whenever possible. Write in the math equivalent of high BBC English, language that is understandable to most anyone that speaks English. One should write, whenever possible, using notation that is clear and that is broadly understood.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ This amateur historian of statistics can supply a pedantic correction that Yule was never a Professor... More interestingly there is a pertinent website at jeff560.tripod.com/stat.html except that it appears to be down at present. $\endgroup$
    – Nick Cox
    Jun 6, 2016 at 17:15
  • 2
    $\begingroup$ math.hawaii.edu/~tom/history/stat.html appears to be a copy. Systematic conventions such as Greek for parameters and roman for variables I understand to be largely due to R.A. Fisher, but there are many hold-outs e.g. $\chi^2$ for sample chi-square statistic shows no signs of fading away. $\endgroup$
    – Nick Cox
    Jun 6, 2016 at 17:22
  • $\begingroup$ @NickCox Fantastic link jeff560.tripod.com/stat.html (up for me...) that references Yule and R.A. Fisher! The earliest mathematical origins of regression obviously go back earlier to Gauss and Laplace, but in my complete amateur searching, they appeared to use different notation. $\endgroup$
    – Matthew Gunn
    Jun 6, 2016 at 17:27
  • $\begingroup$ jeff560.tripod.com/stat.html as I write is a 2014 update; www.math.hawaii.edu/~tom/history/stat.html is a copy of a version from 2007. $\endgroup$
    – Nick Cox
    Jun 7, 2016 at 6:51
  • $\begingroup$ That tripod.com page is still there but much of the historical mathematics terms content has moved to mathshistory.st-andrews.ac.uk/Miller ... if the tripod.com link dies, this particular page is mirrored at mathshistory.st-andrews.ac.uk/Miller/mathsym/stat $\endgroup$
    – Glen_b
    Dec 14, 2021 at 3:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.