Below is the description for the implementation of layer normalization from Stanford's CS 231n:

def layernorm_forward(x, gamma, beta, ln_param):
    Forward pass for layer normalization.

    During both training and test-time, the incoming data is normalized per data-point,
    before being scaled by gamma and beta parameters identical to that of batch normalization.
    Note that in contrast to batch normalization, the behavior during train and test-time for
    layer normalization are identical, and we do not need to keep track of running averages
    of any sort.

    - x: Data of shape (N, D)
    - gamma: Scale parameter of shape (D,)
    - beta: Shift paremeter of shape (D,)
    - ln_param: Dictionary with the following keys:
        - eps: Constant for numeric stability

    Returns a tuple of:
    - out: of shape (N, D)
    - cache: A tuple of values needed in the backward pass

My understanding is that for layer normalization we normalize across rows of the input data, meaning:

For each row $X_i$ consider $\gamma \frac{X_i - mean}{\sqrt{\sigma^2 + eps}} + \beta$. The thing that confused me is that if we are working over rows, it seems that we need $\gamma$ and $\beta$ to be consistent with the number of rows which is $N$ in this case. But it is stated to be $D$, which is the number of columns of the input data, in the description above.


1 Answer 1


N is the batch size. For layer normalization, normalizing across the rows of the input data means that for each data point in the batch (of which there are N), we normalize the vector of values (which is dimension D).

  • $\begingroup$ Is there any reason for that? My intuition is: for batch normalization we want each column to distribute the same so it makes sense to use the same parameter for each column. The I naturally expect layer normalization to force the rows to distribute the same. But if we are using different parameters for different rows that is not the case. $\endgroup$
    – Adam
    Jul 11, 2020 at 1:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.