Below is the description for the implementation of layer normalization from Stanford's CS 231n:
def layernorm_forward(x, gamma, beta, ln_param):
"""
Forward pass for layer normalization.
During both training and test-time, the incoming data is normalized per data-point,
before being scaled by gamma and beta parameters identical to that of batch normalization.
Note that in contrast to batch normalization, the behavior during train and test-time for
layer normalization are identical, and we do not need to keep track of running averages
of any sort.
Input:
- x: Data of shape (N, D)
- gamma: Scale parameter of shape (D,)
- beta: Shift paremeter of shape (D,)
- ln_param: Dictionary with the following keys:
- eps: Constant for numeric stability
Returns a tuple of:
- out: of shape (N, D)
- cache: A tuple of values needed in the backward pass
"""
My understanding is that for layer normalization we normalize across rows of the input data, meaning:
For each row $X_i$ consider $\gamma \frac{X_i - mean}{\sqrt{\sigma^2 + eps}} + \beta$. The thing that confused me is that if we are working over rows, it seems that we need $\gamma$ and $\beta$ to be consistent with the number of rows which is $N$ in this case. But it is stated to be $D$, which is the number of columns of the input data, in the description above.
