Deep learning ... notion of depth

Question

I'm reading from http://www.iro.umontreal.ca/~bengioy/papers/ftml.pdf around page 8.

Here is a quote that I'm not understanding:

"If we include affine operations and their possible composition with sigmoids 
 in the set of computational elements, linear regression and logistic regression 
have depth 1, i.e., have a single level."

Depth or Depth of Architecture refers "to the depth of that graph, i.e. the longest path from an input node to an output node."

I can see why the graph represented in the image has depth of 4. I'm not sure how to think about, say, linear regression in these terms.

Answer 1

Once a linear regresion model is fitted, the output is simply a linear combination of the inputs (the coefficients are the fitted parameters of the regression model, $\hat{\beta}$.) In particular, the model prediction is $y=\hat{\beta}^{T}x$, or if a binary output is desired, we can apply a threshhold:

$y=\left\{ \begin{array}{ll} 1 & \hat{\beta}x \geq c \\ 0 & \hat{\beta}x < c \end{array} \right. $

In logtistic regression, the prediction model is only slightly more complicated. The predicted probability is:

$y=\mbox{logit}^{-1}(\hat{\beta}^{T}x)$

In many cases we make a binary yes or no prediction by applying a threshhold to the above formula. Since $\mbox{logit}^{-1}$ is monotone, this is just a threshhold on $\hat{\beta}x$:

$y=\left\{ \begin{array}{ll} 1 & \hat{\beta}x \geq c \\ 0 & \hat{\beta}x < c \end{array} \right. $

A classical perceptron node computes $\hat{\beta}x$ and applies a threshhold, so a single perceptron node can be used to implement either a linear regression or logistic regression classifier.

Answer 2

Linear regression is of the form $w^T x + b$. "If we include affine operations...in the set of computational elements," then $f(x) = w^T x + b$ is an affine operation, and so linear regression only applies a single "computational element" to the input, and the graph is of depth one.

Similarly, logistic regression is of the form $g(x) = \sigma(w^T x + b)$, where $\sigma(x) = 1 / (1 + e^{-x})$. This as an affine operation composed with a sigmoid, which we're including in the set of computational elements, so by definition $g$ is only applying a single operation to the input $x$ and so it's also of depth 1.