Why the Gaussian Distribution in ML/DL?

Question

So whenever I learn about a method, or a technique to combat some disadvantage present in said method, some mention of transforming data into the Gaussian Distribution exists.

My focus here is mainly on Deep Learning.

For example, when it comes to C.N.Ns, standardizing our input data by subtracting the mean of your input examples by the std of the your input example, (which has the effect of transforming the input feature distributions into a unit gaussian) allows us to not only properly scale our input features but also make it resemble the Gaussian distributions...

(for reasons that are not apparent to me and which I intend to find out in the answers to this question)

Another example is a technique called batch normalization introduced to combat the effect of saturated neurons by being able to place the input value to subsequent layers in an "active" regime of the activation function as to continuously stimulate backprop & learning.

But the question remains.. why Unit Gaussian? Is it because it describes most phenomena out there to the degree that we just simply assume that the each feature (i.e. random variable) in our network should follow the normal distribution because that's the way of life? Or again, is there some underlying theory behind this?

---

EDIT1: Subtracting sample mean and dividing by std does not transform an existing distribution into a Gaussian D. but only applies some scaling and shifting characteristics to the distribution without changing its inherent shape.

Answer 1

Building on whuber's comment, check out the following bit of Python code.

import numpy as np
import matplotlib.pyplot as plt
np.random.seed(2020)
x0 = np.random.normal(0, 1, 1000)
x1 = np.random.normal(11, 5, 1000)
x = np.concatenate([x0, x1])
plt.hist(x)
plt.show()

xbar = np.mean(x)
s = np.std(x, ddof=1)
plt.hist((x-xbar)/s)
plt.show()

You'll notice that your x variable has a very non-Gaussian distribution. Upon subtracting the mean and then dividing by the standard deviation, you graph the same shape, so subtracting the mean and then dividing by the standard deviation does not give you a Gaussian distribution; you have the same kind of shape as you had before, just shifted and compressed (or expanded, if the original standard deviation is less than $1$).

You are correct that the mean of the transformed variable will be $0$ and the standard deviation will be $1$, however, so if your data started out Gaussian-looking, then it will be standard normal-looking after the transformation.