0
$\begingroup$

I'm really struggling to see the analogy between linear regression and a single layer perceptron. They are supposedly the same thing.

I completely understand the concept of the inputs to the neuron being the explanatory variables, and having weights applied to them which are the $\beta$ coefficients, as well as the "bias" term which is the intercept in normal regression terms.

Mostly my problem is that I don't understand 1) whether the data is "fed in" to the neural network one data point at a time or all at once, and 2) whether an iterative method is used to find the $\beta$ coefficients or if it's just least squares. If an iterative method is used, that introduces the whole gradient descent thing. But when people say a single layer neural network and linear regression are the same thing, are they getting into backpropagation at all? Or are they just saying you can "view" regression in that way, and just minimize the loss function in the normal OLS way?

I hope I'm making at least some sense. Any help is really appreciated.

Improve this question
$\endgroup$
5
  • $\begingroup$ “when people say they are the same thing”—What does “they” refer to here? Iterative methods and GD? $\endgroup$
    – Arya McCarthy
    Oct 18, 2021 at 17:35
  • $\begingroup$ Doing gradient descent for linear regression with OLS errors with Newton's method for example produces exactly the closed form estimator, see this answer. $\endgroup$
    – alan ocallaghan
    Oct 18, 2021 at 17:36
  • $\begingroup$ As for 1), I believe this answer varies a lot. Generally no, data isn't fed it one-by-one, but I know that a lot of practitioners use minibatching, where subsets of the data are used for each update of the weights and bias. For 2), it doesn't really matter what method you use to find the coefficients, because you're optimising the same function in either case. Theoretically you could find a closed form estimator for a neural network and use that to find the same solution as using gradient descent; the model is still the same. $\endgroup$
    – alan ocallaghan
    Oct 18, 2021 at 17:40
  • $\begingroup$ They mean that they can configure the network so that it produces exactly the same result as OLS $\endgroup$
    – Aksakal
    Oct 18, 2021 at 17:41
  • $\begingroup$ @AryaMcCarthy edited, basically meant single layer NN = linear regression $\endgroup$
    – fmtcs
    Oct 18, 2021 at 17:52

1 Answer 1

Reset to default
4
$\begingroup$

  • Linear regression is

    $$ y = \mathbf{x} \cdot \boldsymbol{w} + b $$

    that’s exactly the same as a single-layer network with linear activation.

  • Linear regression is usually trained using ordinary least squares, but alternatively you could use an optimization algorithm such as gradient descent.

  • Same as with other neural networks, gradient descent can be used for all data, in mini-batches, or one sample at a time (stochastic gradient descent).

  • It’s just single layer, so you don’t need back propagation.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the answer. So the neural network itself is basically independent of the method of training? Like how the general regression equation $y = \mathbf{x} \cdot \boldsymbol{w} + b$ is independent from the method you choose to estimate the weights? $\endgroup$
    – fmtcs
    Oct 18, 2021 at 17:57
  • $\begingroup$ @fmtcs yes, unless you call it “OLS” because then you mean particular method of training as well. $\endgroup$
    – Tim ♦
    Oct 18, 2021 at 17:58
  • $\begingroup$ Note that if N > p, OLS and gradient descent will find the same answer, though gradient descent will be slow. Batch gradient descent will bounce around the OLS solution, though maybe with some learning rate decay it will converge to OLS solution, I'm not sure. $\endgroup$
    – Jonny Lomond
    Oct 18, 2021 at 18:18
  • 1
    $\begingroup$ @JonnyLomond Yes, by managing the learning rate it is possible to guarantee convergence, for example by shortening steps as necessary to achieve a sufficient reduction in global error on each step. Convergence of Rprop and variants $\endgroup$
    – TMBailey
    Oct 18, 2021 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.