I am trying to visualise the geometry of linear regression with feature representation. I have a regression problem with $n$ data pairs $\mathcal{D}:=\{(\mathbf{x},y)_{i}\}_{i=1}^{n}$, independent variables $\boldsymbol{x}{\,\in\,}\mathcal{X}$, dependent variables $y{\,\in\,}\mathbb{R}$, whose relationship is modelled as (random variables) $Y=f(X)+\varepsilon$, where $\varepsilon\sim\mathcal{N}(0,\sigma^{2}$) and $f$ is an unseen target function. Then the (non-regularised) empirical loss is $\smash{\hat{\mathcal{L}}{\,=\,}\frac{1}{2n}||\boldsymbol{\Phi}\boldsymbol{\beta}-\boldsymbol{y}||^{2}_{2}}$ where design matrix $\smash{\boldsymbol{\Phi}{\,:}{=\,}[\boldsymbol{\phi}(\boldsymbol{x}_{1}),...,\boldsymbol{\phi}(\boldsymbol{x}_{n})]^{\top}{\in\,}\mathbb{R}^{n\times d}}$, dependent variables collected in $\boldsymbol{y}{\,\in\,}\mathbb{R}^{n}$ and the regressor to be found $\boldsymbol{\beta}{\,\in\,}\mathbb{R^{d}}$. Explicit feature mapping $\boldsymbol{\phi}{\,:\,}\mathcal{X}{\,\to\,}\mathcal{F}$ is chosen a priori where $\mathcal{F}$ is some (inner product) feature space with dimensions $\text{dim}(\mathcal{F}){\,=\,}d\,$ i.e. $\boldsymbol{\phi}(\boldsymbol{x}){\,\in\,}\mathcal{F}$ is a vector for some $\boldsymbol{x}{\,\in\,}\mathcal{X}$. The feature representation is introduced because we assume that the target function (evaluated at any $\boldsymbol{x}{\,\in\,}\mathcal{X}$) is approximated by $$f(\boldsymbol{x})\approx\big<\boldsymbol{\beta},\boldsymbol{\phi}(\boldsymbol{x})\big>_{\mathcal{F}}\,\,.$$ Given training data $\mathcal{D}$, the solution to the regression problem is $\hat{\boldsymbol{\beta}}= (\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^{\top}\boldsymbol{y}$ and this implies $$\begin{align}\hat{\boldsymbol{y}}&=\boldsymbol{\Phi}\hat{\boldsymbol{\beta}},\\&=\boldsymbol{\Phi}(\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi})^{-1}\boldsymbol{\Phi}^{\top}\boldsymbol{y},\\&{\,=\,}\boldsymbol{\Pi}\boldsymbol{y},\end{align}$$ where $\boldsymbol{\Pi}{\,:\,}\mathbb{R}^{n}{\,\to\,}\mathbb{R}^{d}$ is a projection matrix that projects vectors onto a subspace $\mathbb{R}^{d}{\,\subseteq\,}\mathbb{R}^{n}$. This solution is valid if the columns of $\boldsymbol{\Phi}$ are linearly independent.
My questions:
Given the above and the fact that the subspace $\mathbb{R}^{d}$ is the column space $\text{col}(\boldsymbol{\Phi})$ (or span of the columns in the design matrix $\boldsymbol{\Phi}$), then is this subspace also the feature space $\mathcal{F}$ (given in terms of the basis (columns) generated by the training data) i.e. is $\mathcal{F}==\mathcal{R}^{d}$ true?
As a bonus but not critical to the answer, perhaps an additional discussion to tie in with the first question. The reason for the first question is that I want to visualise (geometrically) when we choose the feature space to be $\mathcal{F}=\mathcal{H}_{k}$, which is a reproducing kernel Hilbert space (RKHS) associated with kernel $k$. Crudely, an RKHS here can be illustrated as the regressor being a linear combination of features over the training set $\smash{\hat{\boldsymbol{\beta}}=\boldsymbol{\Phi}^{\top}\boldsymbol{\alpha}}$ where $\smash{\hat{\boldsymbol{\alpha}}\in\mathbb{R}^{n}}$ is to be found. I am also aware that the representer theorem defines the expected risk minimisation problem to have a regulariser term, $$\hat{\mathcal{L}}{\,=\,}\frac{1}{2n}\sum_{i=1}^{n}L(h(\boldsymbol{x}_{i}),y_{i}) + \lambda||h||^{2}_{\mathcal{H}},$$ of which the solution is an RKHS function $h(\cdot){\,=\,}\sum_{i=1}^{n}\alpha_{i}k(\cdot,\boldsymbol{x}_{i})$ and $L$ is a squared loss.
