Suppose you want to find $k$ that minimises your cost function $J(k)$. We may want to apply batch gradient descent or stochastic gradient descent. Let's deliberately initialise $k$ with the same number for both BGD and SGD to see the difference in their behavior.
If you apply BGD, this optimisation may look like this:
On the other hand if you apply SGD, this optimisation may look like this:
In both pictures the blue solid curve represents the cost function $J(k)$. But in the second picture there are also dotted curves. In my experiment I used batch size which was $10\text%$ of the whole dataset. So each dotted curve represents the cost function for the current batch. From these dotted curves you can see that the gradient of the batches' cost functions happened to be large multiple times in a row. That's why the point was "pushed" to the deeper minimum.
$\left(*\right)$ As I can see it, if the gradient of most batches' cost functions at a particular point is small enough, then the red point isn't able to get out from the local minimum, precisely because this local minimum makes those gradients small enough. In simple words, that minimum makes the mean gradient of all different batches so small that these batches don't have strength enough to push the point out of this deeper minimum.
Question 1: Was I right in $\left(*\right)$? If the point can't get out from some local minimum, then does it imply that all (or the majority?) these batches have reached their own local minima? Meaning that there is a common local minimum at $k = 0$ in our example. You can see it observing different dotted curves. My line of reasoning is the following one: If most of different batches haven't reached their local minima, then they will eventually push the red point out of this minimum in the picture above (I know it's wrong to say "batches reached their local minima", but I didn't find a better way to express this). So I believe that the point can't get out of the local minimum if this local minimum is the local minimum for the most (how many?) batches.
Question 2: It's not clear to me where does this red point tend to? It obviously can't always tend to a global minimum (which is $k = 2$). My guess is that the point tends to some minimum near the point's initial position that is local minimum for the most of batches (dotted curves). But I'm not sure if I'm right. And what if there is no common minimum for all different batches (dotted curves)?
Question 3: If I'm right in Question 1 and Question 2, then is it the main advantage of SGD? I mean if we use BGD, then it just looks for the nearest local minimum. But if we use SGD, then it looks for the local minimum that is going to be the local minimum for most of the batches? It implies then that this latter local minimum reduced most batches' gradients enough not to escape this local minimum.
Question 4: So I believe that BGD answers the question: "Where should the red point go to reduce the overall error?". And SGD answers the question: "Where should the red point go to reduce the current batch error?". Then if most of the batches says: "Go left", then the red point gets out from the local minimum and moves to the left and, therefore reduces the overall error even more (because most of the batches said to go left). Is that right?
If possible, elaborate on this a little bit so that I can build the intuition. Thank you in advance!
As a side note, $J(k) = \frac 1n\sum\left(\sin\left(kX\right) - Y\right)^2$, where $X$ is a single feature vector and $Y$ is a vector of true answers and $k$ is a coefficient we want to find so that $J(k)$ was minimal.