I have a feedforward neural network $F(W): \mathbb R^d \rightarrow \mathbb R^k$ with $Relu$ activation, $m$ neurones per layer, $L$ layers and softmax on the output layer. $W$ denotes the weight matrices.

My training data are points concentrated in compact and connected manifolds $M_i \in \mathbb R^d$. Each manifold has diameter at most $D$, which means that if $x_1, x_2 \in M$, the distance between $x_1$ and $x_2$ is at most $D$. Let's consider $D << 1$.

When I run gradient descent to minimise the loss function $L(W,x_1) =\frac 1 2 \| F(W,x_1) - y \|^2$, I know that $L(x_1)$ decreases by $\eta \nabla L(x_1)$, where $\eta$ is the step size.

Assuming that $x_1$ and $x_2$ belong to the same manifold $M$, what can I say about $L(W,x_2)$? does it decrease when I run gradient descent on $L(W,x_1)$?

In other words, how does the change of weights given by one iteration of gradient descent on $L(W,x_1)$ affects the loss function computed on points close to $x_1$?

Is it reasonable to say that $L(W,x_2)$ is expected to decrease as well or not?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.