In machine learning, why are superscripts used instead of subscripts? - Cross Validated most recent 30 from stats.stackexchange.com 2019-08-21T00:46:15Z https://stats.stackexchange.com/feeds/question/193908 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/193908 20 In machine learning, why are superscripts used instead of subscripts? entpnerd https://stats.stackexchange.com/users/102930 2016-02-03T21:13:00Z 2016-02-05T13:34:32Z <p>I'm taking <a href="https://www.coursera.org/learn/machine-learning">Andrew Ng's course on Machine Learning through Coursera</a>. For equations, superscripts are used instead of subscripts. For example, in the following equation $x^{(i)}$ is used instead of $x_i$:</p> <p>$J(\theta_0, \theta_1) = \frac{1}{2m} \sum\limits_{i=1}^{m}{(h_\theta(x^{(i)}) - y^{(i)})^2}$</p> <p>Apparently, this is common practice. My question is why use superscripts instead of subscripts? Superscripts are already used for exponentiation. Granted I seem to be able to disambiguate between the superscript and exponentiation use cases by paying attention to whether or not parentheses are present, but it still seems confusing.</p> https://stats.stackexchange.com/questions/193908/-/193915#193915 11 Answer by Cliff AB for In machine learning, why are superscripts used instead of subscripts? Cliff AB https://stats.stackexchange.com/users/76981 2016-02-03T21:52:09Z 2016-02-04T18:54:34Z <p>The use of super scripts as you have stated I believe is <em>not</em> very common in machine learning literature. I'd have to review Ng's course notes to confirm, but <strong>if</strong> he's putting that use there, I would say he would be origin of the proliferation of this notation. This is a possibility. Either way, not to be too unkind, but I don't think many of the online course students are publishing literature on machine learning, so this notation is not very common in the actual literature. After all, these are introductory courses in machine learning, not PhD level courses. </p> <p>What is very common with super scripts is to denote the iteration of an algorithm using super scripts. For example, you could write an iteration of Newton's method as </p> <p>$\theta^{(t+1)} = \theta^{(t)} - H(\theta^{(t)}) ^{-1} \nabla \theta^{(t)}$</p> <p>where $H(\theta^{(t)})$ is the Hessian and $\nabla \theta^{(t)}$ is the gradient. </p> <p>(...yes this is not quite the best way to implement Newton's method due to the inversion of the Hessian matrix...)</p> <p>Here, $\theta^{(t)}$ represents the value of $\theta$ in the $t^{th}$ iteration. This is the most common (but certainly not only) use of super scripts that I am aware of. </p> <p><strong>EDIT:</strong> To clarify, in the original question, it appeared to suggest that in the ML notation, $x^{(i)}$ was equivalent to statistic's $x_i$ notation. In my answer, I state that this is not truly prevalent in ML literature. This is true. However, as pointed out by @amoeba, there is plenty of superscript notation in ML literature for data, but in these cases $x^{(i)}$ does not typically mean the $i^{th}$ observation of a single vector $x$. </p> https://stats.stackexchange.com/questions/193908/-/193925#193925 26 Answer by amoeba for In machine learning, why are superscripts used instead of subscripts? amoeba https://stats.stackexchange.com/users/28666 2016-02-04T00:11:58Z 2016-02-04T10:12:03Z <p>If $x$ denotes a vector $x \in \mathbb R^m$ then $x_i$ is a standard notation for the $i$-th coordinate of $x$, i.e. $$x = (x_1, x_2, \ldots, x_m)\in\mathbb R^m.$$</p> <p>If you have a collection of $n$ such vectors, how would you denote an $i$-th vector? You cannot write $x_i$, this has other standard meaning. So sometimes people write $x^{(i)}$ and that is I believe why Andrew Ng does it. </p> <p>I.e.</p> <p>\begin{equation} x^{(1)} = (x_1^{(1)}, x_2^{(1)}, \ldots, x_m^{(1)}) \in \mathbb R^m\\ x^{(2)} = (x_1^{(2)}, x_2^{(2)}, \ldots, x_m^{(2)}) \in \mathbb R^m\\ \ldots \\ x^{(n)} = (x_1^{(n)}, x_2^{(n)}, \ldots, x_m^{(n)}) \in \mathbb R^m.\\ \end{equation}</p> https://stats.stackexchange.com/questions/193908/-/194074#194074 4 Answer by Aksakal for In machine learning, why are superscripts used instead of subscripts? Aksakal https://stats.stackexchange.com/users/36041 2016-02-04T19:16:21Z 2016-02-05T13:34:32Z <blockquote> <p>Superscripts are already used for exponentiation. </p> </blockquote> <p>In mathematics superscripts are used left and right depending on the field. The choice is always historical legacy, nothing more. Whoever first got into the field set the convention of using sub- or superscripts.</p> <p>Two examples. Superscripts are used to denote derivatives: $f(x)^{(n)}$</p> <p>In <a href="https://en.wikipedia.org/wiki/Tensor" rel="nofollow">tensor</a> algebra both super and subscripts are used heavily for the same thing like $R^i_i$ could mean $i$ rows and $j$ columns. It's quite expressive: $T_i^k=R_i^jC_j^k$</p> <p>Also I remember using scripts before letters (prescripts) in Physics, e.g. $^i_jB_k^l$. I think it was with tensors.</p> <p>Hence, the choice of superscripts by Ng is purely historical too. There's no real reason to use or not use them, or prefer them to subscripts. Actually, I believe that here ML people are using tensor notation. They definitely are well versed in the subject, e.g. see <a href="http://arxiv.org/abs/1509.05009" rel="nofollow">this</a> paper.</p>