Which weights are the weights of parameters of Linear Regression? 
I have just started studying Linear Regression using Neural Network and I believe this is a naive question.
I have a neural network for a Linear Regression Model with 3 independent variables. I have 1 hidden layer with 3 neurons with linear activations for each neuron. The last neuron is just another linear activation function that sums the three outputs from the hidden layer and compare the predicted value with the actual value.
I am wondering which weights are the final weights between the weights before the hidden layer and the weights after the hidden layer that I can use for the parameters of the Linear Regression Model. Hope to hear some explanations.
 A: A Neural network with one layer using linear activation is a linear regression. If you have a multi-layer neural network that only uses linear activations, it will reduce to linear regression. In other cases, it is a different algorithm. The same applies to logistic regression: it is the simplest, single-layer neural network with sigmoid activation, but if you add more layers, it is a different algorithm. You cannot use neural networks to "find" parameters of linear or logistic regression unless you mean the special cases mentioned above.
A: This neural network does regression, but what it does is not equivalent to linear regression with the original features. So, you cannot just get the weights of either layer and build a linear regression model with those.
In order to emulate linear regression with a neural network, you should not have a hidden layer, but just one output layer.
A: Since your network has linear activation functions it can be written as a linear regression (it couldn't be if it used other activation functions).  If write $w_{1i}$ for the weight of $x_i$ in neuron $h_1$ and $v_i$ for the weight of the output of $h_i$ in the final output, and $\beta_i$ for the coefficient of $x_i$ in a linear regression, then
$$\beta_1=\sum_{j=1}^3 v_jw_{j1}$$
$$\beta_2=\sum_{j=1}^3 v_jw_{j2}$$
$$\beta_3=\sum_{j=1}^3 v_jw_{j3}$$
The estimates will be the same as the OLS estimates if you use in-sample squared error as your loss function and if you actually find the globally optimal $w$ and $v$. I don't know whether back-propagation guarantees finding the global optimum in this very simple case; it doesn't in more complicated cases.
