In this article the author is looking at dropout training and trying to show it is equivalent in some way to adding a penalty term to the loss function.

On page 5, in the little section called "Linear Regression" he comes to a penalty term given by

$$\newcommand{\bbeta}{\boldsymbol{\beta}} R(\bbeta) \propto \bbeta^T \operatorname{diag}(X^TX)\bbeta = \beta_1^2 \left( \left(x^{(1)}_{1}\right)^2 + \dotsb + \left(x^{(n)}_{1}\right)^2 \right) + \dotsb + \beta_d^2\left( \left(x^{(1)}_{d}\right)^2 + \dotsb + \left(x^{(n)}_{d}\right)^2 \right)$$

Here $\boldsymbol{\beta} \in \mathbb{R}^d$ is the weights of the single layer and $X$ is the design matrix whose rows are the training examples $\mathbf{x}^{(1)}, \dotsc, \mathbf{x}^{(n)} \in \mathbb{R}^d$.

He describes this penalty term as

...a form of ridge regression where each column of the design matrix is normalized before applying the $L^2$ penalty.

Question: What does this mean? I understand the general notion of normalizing here is that various entries of the training vectors may have different scales, so their contribution to the loss may be disproportionate. And I guess the general idea is that if the first entry of the training vectors has big scale, and the second one has small scale, then we should penalize the size of $\beta_1$ more than we penalize $\beta_2$? Why?

Most of all, how does putting $\left( \left(x^{(1)}_{1}\right)^2 + \dotsb + \left(x^{(n)}_{1}\right)\right)^2$ into the penalty term for the size of $\boldsymbol{\beta}$ correspond to normalizing the design matrix? (I guess normalizing the design matrix would be dividing column $i$ of $X$ by $\sqrt{\left( \left(x^{(1)}_{i}\right)^2 + \dotsb + \left(x^{(n)}_{i}\right)\right)^2}$.)

  • 1
    $\begingroup$ @CagdasOzgenc When you say "ridge regression with input vectors normalized...", what would it look like with them unnormalized? Just $\boldsymbol{\beta}^T\boldsymbol{\beta}$? Then how do the feature vectors get involved at all? $\endgroup$
    – Eric Auld
    Jun 16 '18 at 18:22

If the columns of $X$ have mean zero, then their variances are $\sigma^2 = \text{diag}(X^TX)$. Then if $X_s$ is the scaled version of $X$ so that $X = X_s \cdot \text{diag}(\sigma)$, the standard ridge loss can be written as

$$ \begin{align} \|y - X_s\beta_s\|^2 + \lambda \|\beta_s\|^2 &= \|y - X_s \cdot \text{diag}(\sigma) \cdot \frac{1}{\text{diag}(\sigma)} \beta_s \|^2 + \lambda \|\text{diag}(\sigma) \cdot \frac{1}{\text{diag}(\sigma)} \beta_s\|^2 \\ &= \|y - X\beta\|^2 + \lambda\|\text{diag}(\sigma) \beta\|^2 \end{align} $$

where $\beta = \frac{1}{\text{diag}(\sigma)} \beta_s$.

TL;DR The $\text{diag}(X^TX)$ are the coefficients you use to standardize the matrix, and you can shift them from dividing through the matrix to multiplying through the weights.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.