2
$\begingroup$

What notation is used to represent the normalised or z-score value of a vector $x$?

Or, given:

$? = \dfrac {x - \mu } {\sqrt{\sigma^2}}$

Where $\mu$ and $\sigma$ are calculated across the whole population of $x$.

Is there a canonical, agreed or widely used symbol to replace "$?$"?

My specific context is batch normalisation. If the vector $z = Wa + b$ is the biased, weighted sum of a layer in a neural network, what notation would I use to represent that vector after it has been normalised?

It may be a little confusing since $z$ is a commonly used variable for this pre-activation weighted sum, and I want the notation to represent the z-score version of $z$.)

Is the answer the same in the context of scalars, vectors and matrices?

A similar question exists for physics, but I would like to know the answer from the machine learning perspective.

Improve this question
$\endgroup$
5
  • $\begingroup$ This looks like it could potentially be reopened but I'd like to see some changes first for clarification. One thing that would help a lot is to show an example of usage with the "plus epsilon". Also it might help to clarify the title as what you are asking is a little different from "the normalised value of $x$" in the conventional sense - the adjustment here seems to be important, otherwise it would be as simple as everyone agreeing it was a z-score (or perhaps a t-score depending on how the standard deviation was found) $\endgroup$
    – Silverfish
    May 4, 2018 at 18:18
  • $\begingroup$ I removed the epsilon as it was a distraction to the question, and gave context to further clarify. $\endgroup$
    – Tom Hale
    May 5, 2018 at 11:04
  • $\begingroup$ Thanks, I voted to reopen. If you want the answer specifically in the context of neural nets, perhaps add that tag and maybe include it in the title to the question? Anybody who's been studying statistics quickly becomes aware that notation and even terminology can vary between different sub-fields so if you want to know the term within a particular context it helps to be as specific as possible! $\endgroup$
    – Silverfish
    May 5, 2018 at 13:39
  • $\begingroup$ It may also help to clarify whether the $\mu$ and $\sigma$ are in fact true population parameters, or merely estimates thereof. $\endgroup$
    – Silverfish
    May 5, 2018 at 13:40
  • $\begingroup$ @Silverfish thanks, I've updated based on both your comments. $\endgroup$
    – Tom Hale
    May 6, 2018 at 6:10

1 Answer 1

Reset to default
1
$\begingroup$

Without the $\varepsilon$ this would be a standard score aka $z$-score. So,

$$ z_{\varepsilon} = \frac{x-\mu}{\sqrt{\sigma^2 + \varepsilon}} $$

would seem appropriate.

In some statistical/mathematical texts vectors and matrices are denoted in bold font. For vectors that would be, e.g. $\mathbf{x}$, $\mathbf{\mu}$, $\mathbf{z}_{\varepsilon}$. Matrices are then denoted as bold font capital letters: $\mathbf{X}$, $\mathbf{M}$, $\mathbf{Z}_{\varepsilon}$.

In physics an overset arrow (or "harpoon") is common for vectors: $\overset{\rightharpoonup}{x}$, $\overset{\rightharpoonup}{\mu}$, $\overset{\rightharpoonup}{{z}_{\varepsilon}}$. (A little shorter/simpler is with \vec{}: $\vec{x}$, $\vec{\mu}$, $\vec{z}_{\varepsilon}$.)

A common way to denote transformed variables is simply with a superscript asterisk: $x^{*}$, $y^{*}$.

Overset bar ($\bar{x}$), tilde ($\tilde{x}$) and hat ($\hat{x}$) should be considered reserved symbols, denoting respectively: mean, median, and prediction.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks! The $\varepsilon$ is added to the denominator to prevent division by zero. The issue is that $z$ is already commonly used in machine learning to represent the weighted sum of the input vectors to a neuron (or layer), with a bias term added. I've seen $\hat x$ used by Andrew Ng... that does seem to conflict with the mean, but does resonate nicely with the standardisation of a unit vector. $\endgroup$
    – Tom Hale
    May 1, 2018 at 5:05
  • $\begingroup$ I'm going with $\overset z z$ ($\overset z z$) $\endgroup$
    – Tom Hale
    May 3, 2018 at 11:43
  • 1
    $\begingroup$ Updated now. Personally, I'd go with z^* = $z^*$. $\endgroup$
    – Jim
    May 3, 2018 at 12:24
  • $\begingroup$ I like that that $z^*$ shows a new value of $z$, but sadly it doesn't say that the newness is due to the z-score. $\endgroup$
    – Tom Hale
    May 6, 2018 at 6:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.