In more classical statistical methods like linear regression, we can quantify how well our model generalizes under certain strong assumptions. For example, we know that $\hat Y = X \hat \beta \sim \mathcal{N}(X\beta, \sigma^2 H)$ assuming that $Y\sim \mathcal{N}(X \beta, \sigma^2)$.
In machine learning theory, the approach is completely different: We do not make any assumptions on the (conditional) distribution and use tools like VC Dimensions to derive bounds on the difference $\sup_{h \in \mathcal{H}} |L_S(h) - L(h)|$ for a given function class $\mathcal{H}$ with $L_S(h)$ and $L(h)$ denoting the empirical and the "true" error of a hypothesis $h \in \mathcal{H}$. This then allows us to bound $L(\hat h)$ where $\hat h$ is our fitted model. These bounds are obviously very loose.
My question: Is there a theory bridging this gap between "no assumptions, mostly loose bounds" and "strong assumptions, very good bounds"?
I am aware of no free lunch. However, in most applications of machine learning we have real-world distributions (whatever that exactly means) that seem to be quite nicely behaved in the sense that our model generalizes better than we would necessarily expect from learning theory.
