Backpropagation wrong? Doesn't it update dependent variables in hidden layer

Question

In a multi layer perceptron or feedforward neural network, isn't backpropagation updating weights of the middle layers that are dependent variables? So for a particular hidden layer, it calculates all of the partial derivatives of all of the nodes and updates the weights by a learning rate/optimizer. But isn't it updating the partial derivatives of functions that are dependent variables? So if all the input values to that layer are fixed, then updating all of the partial derivatives of the hidden node is correct because that is the best thing to do to decrease the loss. But we are updating the previous layer(s) as well, so the input values for the particular hidden node after the previous layer(s) are updated, are no longer the same values so it updated the weights of the particular hidden node on input values that are no longer used.

Do anyone have insight on this? Maybe I'm totally off?

Thanks so much for the help.

Answer 1

Parameters and dependent variables are completely different concepts.

Dependent variables are the target that you want to measure as a function of your independent variable. For example, if you have information about a person's height and weight, you might try to make an inference about the person's gender (classification) or age (regression).

Parameters are the quantity that you optimize to make better predictions. In a neural network, this is usually the composition of several functions ("layers"). For example, a neural network with one hidden layer predicts the dependent variable using $f(A(f(Bx+c)+d)$ where $x$ is your independent variable (feature vector), and $f$ the activation function and $A, B, c, d$ are all parameters of appropriate shape (dimension).

Special cases of neural networks with no hidden layers correspond to different generalized-linear-models. For example, if the activation $f$ is the softmax function, the result is a logistic regression. If $f$ is the identity function, the result is OLS. The independent and dependent variables are the same: inputs are independent variables, outputs are dependent variables. Hopefully this gives you some intuition to unify regression terminology (independent & dependent variables) with neural network terminology.