Difference between linear regression and neural network

I am obviously confused with terms, and different concepts behind it. Each websites gives different intuitions. With all intuitions my brain is full of confusion now. Please help me to address what is right.

1. Neural Network = Multi Layer Perceptron
2. Linear Network/Regression = Neural Network ( with No hidden layer) only input and output layer.

This Link proves linear regression without hidden layer.

Now the confusion is with respect to binary output and continuous value?

Can I claim below points are also called Linear Regression?

1. The neural network with binary output with one or more hidden layers.

2. Some site claims linear regression means the continuous value output. If I have an MLP with hidden layers, and its output is continuous value (ex: house price), then is it called linear regression?

3. Neural Network with linear activation functions ( doesn't matter binary output, continuous output value, hidden layer)

Hope you understood my confusion.

• Why would you like to call neural network linear regression, or the other way around?
– Tim
Nov 8 '18 at 15:14
• @Tim, I mean different people in community tells like this. I am not sure if it is right or wrong?. :) Nov 8 '18 at 15:29
• I edited your title to be more informative, feel free to revert the change or edit it further if you wish
– Tim
Nov 8 '18 at 16:05
• Sure, Thanks for your wonderful reply. Sounds it take some time to understand it completely.(All the links you provided. :) Nov 9 '18 at 4:54

Linear regression is defined in terms of a linear function:

$$\hat y = w_1 x_1 + w_2 x_2 + \dots + w_k x_k + b$$

where $$\hat y$$ is the prediction for the target variable $$y$$, $$x_1,x_2,\dots,x_k$$ are the features, $$w_1,w_2,\dots,w_k$$ are the weights and $$b$$ is a bias term.

In terms of neural networks, you have multiple inputs $$x_1,x_2,\dots,x_k$$ and a single output $$\hat y$$. So there is a single densely-connected layer, a single output layer with one unit and no hidden layer. There is also no activation (or link function), as with the link function we would rather be talking about generalized linear models (e.g. logistic regression). But there is more in the definition of GLMs than just the link function. Linear regression means also using squared loss and no regularization.

Using linear regression for predicting binary outputs is a suboptimal choice, same for counts, and there are specialized GLMs for many different problems. So basically yes, we define and use linear regression for continuous outputs.

All this said, I don't really think that calling linear regression a neural network makes much sense. Sure, there are people who will call it a neural network or even artificial intelligence, but unless you want to monetize your startup and need to make much marketing fuss around the big data blockchain deep learning artificial intelligence that you do, it would sound rather ridiculous.

1. The neural network with binary output with one or more hidden layers - No. Because Linear Regression = Input matrix * Weight Matrix = Output Score. Linear regression will have no hidden layers. When this output score is subject to a step up activation function or a threshold then we are getting into linear binary classification.
2. Some site claims linear regression means the continuous value output. If I have an MLP with hidden layers, and its output is continuous value (ex: house price), then is it called linear regression? This is tricky and conceptualization is important. ANN can solve regression problem that is problems with continuous outcome variable. ANN is a flexible and complex algorithm. It can dynamically pick the best regression model, be it linear, logistic, or polynomial and if the prediction is not accurate enough it has hidden layers at its arsenal to boost the prediction power with higher accuracy. So, ANN can do the regression job. However, linear regression works best only when the linear regression equation is the best fit to the data available. This is the difference.
3. Neural Network with linear activation functions ( doesn't matter binary output, continuous output value, hidden layer) See, when you have linear activation function, it turns all layers into one as the linear combination of all layers with be a linear, thereby reducing it to an input output linear function which is a nothing but linear regression. On a quick note, a single hidden layer with a sigmoid activation function and may be in combination with a step up function is a logistic regression.
Hope this answers all your doubts and confusion.