# Linear regression using two different classes of basis functions

Let's say we have 1D data $$Y = \{ y_i \mid y_i \in \mathbb{R} \}$$, and regressors $$X = \{ x_i \mid x_i \in \mathbb{R} \}$$, and we try doing basis regression.

Suppose we find that we can perfectly describe $$Y$$ as being linear with respect to two different classes of basis functions: $$Y = \Phi_1(X)\beta_1$$ and $$Y = \Phi_2(X)\beta_2$$. (For a concrete example, suppose $$\Phi_1(X) \in \mathbb{R}^{N \times K}$$ comes from the responses of $$K$$ radial basis functions, while $$\Phi_2(X) \in \mathbb{R}^{N \times K}$$ is from $$K$$ cosine basis functions.)

Can we conclude anything about $$\Phi_1(X)$$ and $$\Phi_2(X)$$? I wouldn't have thought so, but empirically, it seems like you can write $$\Phi_1(X) \approx \Phi_2(X) A + \mu$$, with $$A \in \mathbb{R}^{K \times K}$$. In other words, the bases are linear transformations of one another.

Is this always true? This seems to suggest that if you can write some signal $$y$$ as being linear in some feature space, there is only one such feature space (up to linear transformations). Counterexamples would be very helpful!

• All you can conclude is that a given point (namely $Y$) is in both the column span of $\Phi_1(X)$ and the column span of $\Phi_2(X)$. OK, so we have two (linear) sets, and there exists a point they have in common. We cannot conclude the sets are linear transformations of each other.
– Ben
Commented Jul 21, 2022 at 17:42
• Thanks, I see that now! (I think part of my issue was that, when simulating, the different basis functions I was choosing were not as orthogonal to one another as I was assuming.) Commented Jul 21, 2022 at 17:49
• What you can conclude is that when all $Y$ of the form $\Phi_1(X)\beta$ can be expressed as linear combinations of $\Phi_2(X),$ then $\Phi_1(X)$ is a linear transformation of $\Phi_2(X).$ (But that's a trivial observation...)
– whuber
Commented Jul 21, 2022 at 18:07

I think you have an easy counter-example with $$X= y$$ and $$\Phi_i$$ as the identity in the first coordinate and whatever else on other coordinates, taking $$\beta_1 = \beta_2 = (1, 0, ..., 0)$$
• Hi, thanks for your reply! Sorry, I realized there was a mis-specification in my original post. The regressors $X$ are also 1D, it's only K-dimensional in the basis space. Commented Jul 21, 2022 at 17:21
• Ah, I see. I don't think "whatever else" is quite enough, but I get the idea, thank you! You specifically need the remaining bases in $\Phi_1$ and $\Phi_2$ to be orthogonal to each other. So a specific example would be $y = \cos(X)$, $\Phi_1(X) = [\cos(X); \cos(2X)]$ and $\Phi_2(X) = [\cos(X); \sin(X)]$. Commented Jul 21, 2022 at 17:44
• Why $\Phi_1$ and $\Phi_2$ should be orthogonal ? Following your example, if $\Psi_i(X) = [\cos(X), f_i(X)]$ and $\beta_1 = \beta_2 = (1, 0)$ ? Commented Jul 21, 2022 at 17:47