Say I have a neural network with some unknown number $N$ of hidden layers. Assume I know the structure (e.g. feedforward, convolutional, or recurrent) of the first $k$ of these hidden layers but know nothing about the remaining $N-k$ layers. (I also don't know $N$.) Assuming the weights in the unknown part are fixed, then if I know the output generated by the full network, is it possible to use the loss function (e.g. MSE) to train only the hidden layers for which I know the structure without knowing anything about the unknown layers?
$\begingroup$
$\endgroup$
2
asked Apr 15, 2019 at 2:04
-
$\begingroup$ The data flows from thr input layers to the output layers. Means, that the data flows in a forward direction across the hidden layers ( Forward Propogation ). When we calculate the gradient of the loss function, we propogate it backwards in a backward direction ( Back propogation). For optimizing a layer, we require gradients from the next layer. If we don't know the ( k + 1 )th layer we cannot optimize the ( k )th layer. $\endgroup$– user234584Commented Apr 15, 2019 at 6:28
-
$\begingroup$ typically you do the other way around, which is called transfer learning. $\endgroup$– gunesCommented Apr 15, 2019 at 10:09
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You are facing the issue that you have a composition of two functions, $f(\cdot)$ being your known part of the neural network and $g(\cdot)$ being the unknown part. The output is then $g(f(x))$ and there is some loss function $\mathcal{L}(g(f(x)), y)$ comparing the network output to the correct answer. To train the network, you need to estimate how much each of the weights contributed to the error, which is done by computing the derivative $\frac{\partial \mathcal L}{\partial w}$.
Since you don't know the analytical form of $g(\cdot)$, you cannot analytically compute this derivative. That leaves you with two options:
Numerical differentiation: instead of computing the derivative analytically, you can perturb a weight $w$ by a small amount and see how it affects $\mathcal L$. Note that if you have many weights, this approach will be extremely slow.
Approximate $g(\cdot)$: If you can evaluate $g(\cdot)$, nothing prevents you from creating another neural network $\hat g(\cdot)$ and train it to mimic $g$ simply by feeding random inputs to $g$ and training $\hat g$ to predict the same thing. Then you can use this new neural network as a surrogate for training $f$ which will allow you to evaluate the gradients analytically.
Either way is somewhat impractical.
answered Apr 15, 2019 at 10:20