Does the n >> p holds also for minibatches It's pretty known that when dealing with models (without regularization) the main assumption is $n >> p$ where $p$ is the number of features in the dataset
Let's suppose that we have 1.000.000 samples with 500 features. Then, the closed form of the perceptron/linear regression is valid since $X$ is most likely invertible.
However, calculating the inverse and the matrix multiplication might be very expensive in this case, so suppose that we than approach the problem from a numerical optimization approximation using minibatch SGD (for example as we would do with Keras)
It's known that there is no magic trick to find the size of the minibatch, however I would like to know if, in order to have good approximation, the $n >>p$ should hold also for the minibatch, so that the solutions estimated on the minibatches are relatively close one to the other, and so it's able to lead to the solution
 A: I believe that this question arises from a misunderstanding about how neural networks are estimated compared to regression models. Consider the example of an OLS regression. In the OLS regression, the parameters can be estimated in one step using the normal equations $\hat \beta = (X^T X)^{-1}X^T y$ (but in practice, SVD or QR decompositions are used to improve precision). The normal equations (or QR, or SVD) require a certain matrix to be full-rank matrix so that the regression coefficients are identified.
These facts about OLS regression are not relevant to a neural network because training a neural network with gradient descent mini-batching does not involve solving a linear system involving the design matrix $X$. The gradient descent update of the model parameters $\theta$ has the form
$$
\theta^{(i+1)} =\theta^{(i)} - \eta\nabla L( f(X),y | \theta^{(i)})
$$
where $\eta$ is the learning rate and $ L( f(X),y | \theta^{(i)})$ is the loss for the mini-batch computed with the current parameters $\theta^{(i)}$ and $f$ is the network's predictions.
For the common NN losses that I'm familiar with, there is no component of computing the gradient which requires inverting a matrix formed from $X$.
A common scenario for mini-batch SGD is using a single sample $n=1$ in each mini-batch. If it were true that mini-batch gradient descent required $n > p$, then these neural networks would be restricted to only utilizing $p = 0 < 1 = n$ features, which is clearly a very restrictive requirement!

As an aside, you do not have to have $n > p$ to estimate a regression (even without regularization). As an example, you can directly apply (mini-batch or batch) gradient descent to a rank-deficient design matrix, and you will be able to estimate coefficients. However, you are giving up identifiability because there are many vectors of coefficients that obtain the least loss in this setting, and this procedure only finds one such vector.
