I have a feedforward neural network $F(W): \mathbb R^d \rightarrow \mathbb R^k$ with $Relu$ activation, $m$ neurones per layer, $L$ layers and softmax on the output layer. $W$ denotes the weight matrices.
My training data are points concentrated in compact and connected manifolds $M_i \in \mathbb R^d$. Each manifold has diameter at most $D$, which means that if $x_1, x_2 \in M$, the distance between $x_1$ and $x_2$ is at most $D$. Let's consider $D << 1$.
When I run gradient descent to minimise the loss function $L(W,x_1) =\frac 1 2 \| F(W,x_1) - y \|^2$, I know that $L(x_1)$ decreases by $\eta \nabla L(x_1)$, where $\eta$ is the step size.
Assuming that $x_1$ and $x_2$ belong to the same manifold $M$, what can I say about $L(W,x_2)$? does it decrease when I run gradient descent on $L(W,x_1)$?
In other words, how does the change of weights given by one iteration of gradient descent on $L(W,x_1)$ affects the loss function computed on points close to $x_1$?
Is it reasonable to say that $L(W,x_2)$ is expected to decrease as well or not?