I do understand that BN does work; I just don't understand how "fixing" (changing) the distribution of every mini-batch doesn't throw everything completely out of whack.
For example, let's say you were (for some reason) training a network to match a letter to a number grade. You have one feature - the numeric value.
The batch input to one of your non-linear layers is
(90, A), (80, B), (70, C). It gets normalized first, and the B data point becomes
(0, B) - ignoring the constant factor/bias parameters since that's basically just another linear layer.
Your second training batch input to that layer is
(60, D), (60, D), (60, D). These get normalized as well.
Now you have 3 data points that say
(0, D) - but your network just spent time learning that a value of "0" at this point in the network should tend towards "B"
How does this sort of transformation not break down training in its entirety?
Even though you likely normalized your features before-hand, aren't mini-batches going to end up having significantly differing means and variances that end up throwing everything off?
A feature being equal to 0.1 in one batch might mean something completely different than that feature being equal to 0.1 in another.
What's my fundamental misunderstanding here? Apologies if this is a silly question; I haven't been able to find any answer to this on the web, since I'm assuming it's more of my failure to understand statistics than BN specifically.