3
$\begingroup$

I'm reading from http://www.iro.umontreal.ca/~bengioy/papers/ftml.pdf around page 8.

Here is a quote that I'm not understanding:

"If we include affine operations and their possible composition with sigmoids 
 in the set of computational elements, linear regression and logistic regression 
have depth 1, i.e., have a single level."

Depth or Depth of Architecture refers "to the depth of that graph, i.e. the longest path from an input node to an output node."Here is a simple example demonstrating a depth of 4

I can see why the graph represented in the image has depth of 4. I'm not sure how to think about, say, linear regression in these terms.

$\endgroup$

2 Answers 2

1
$\begingroup$

Once a linear regresion model is fitted, the output is simply a linear combination of the inputs (the coefficients are the fitted parameters of the regression model, $\hat{\beta}$.) In particular, the model prediction is $y=\hat{\beta}^{T}x$, or if a binary output is desired, we can apply a threshhold:

$y=\left\{ \begin{array}{ll} 1 & \hat{\beta}x \geq c \\ 0 & \hat{\beta}x < c \end{array} \right. $

In logtistic regression, the prediction model is only slightly more complicated. The predicted probability is:

$y=\mbox{logit}^{-1}(\hat{\beta}^{T}x)$

In many cases we make a binary yes or no prediction by applying a threshhold to the above formula. Since $\mbox{logit}^{-1}$ is monotone, this is just a threshhold on $\hat{\beta}x$:

$y=\left\{ \begin{array}{ll} 1 & \hat{\beta}x \geq c \\ 0 & \hat{\beta}x < c \end{array} \right. $

A classical perceptron node computes $\hat{\beta}x$ and applies a threshhold, so a single perceptron node can be used to implement either a linear regression or logistic regression classifier.

$\endgroup$
2
$\begingroup$

Linear regression is of the form $w^T x + b$. "If we include affine operations...in the set of computational elements," then $f(x) = w^T x + b$ is an affine operation, and so linear regression only applies a single "computational element" to the input, and the graph is of depth one.

Similarly, logistic regression is of the form $g(x) = \sigma(w^T x + b)$, where $\sigma(x) = 1 / (1 + e^{-x})$. This as an affine operation composed with a sigmoid, which we're including in the set of computational elements, so by definition $g$ is only applying a single operation to the input $x$ and so it's also of depth 1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.