1
$\begingroup$

I want to calculate the Coefficient of Multiple Correlation $R^2$ for a multiple linear regression with polynomial features of degree >= 2 (with interaction terms).

Let's say I want to obtain the regression polynomial of degree 3 of the the relation $y\left(x_1, x_2, x_3\right)$. Using f.i. Elastic Net regression, the result looks like:

$$y = c_0 + c_1 x_1 + c_2 x_2 + c_3 x_3 + c_4 x_1^3 + c_5 x_1^2 x_3 + c_6 x_2 x_3$$

Several coefficients have alread been set to 0, thus only a selected set of variables remains.

Now I want to calculate the Coefficient of Multiple Correlation $R^2$, which is defined as follows: $$R^2 = c_{y,x}^T R_{x,x}^{-1} c_{y,x}$$ with $c_{y,x} = \mathbb{Corr}(\mathbf{y},\mathbf{x}_i)$ and $R_{x,x} = \mathbb{Corr}(\mathbf{x}_i,\mathbf{x}_j)$:

$$\boldsymbol{c}_{\mathbf{y},\mathbf{x}} = \begin{bmatrix} r_1 \\ r_2 \\ \vdots \\ r_m \end{bmatrix} \quad \quad \quad \boldsymbol{R}_{\mathbf{x},\mathbf{x}} = \begin{bmatrix} r_{1,1} & r_{1,2} & \cdots & r_{1,m} \\ r_{2,1} & r_{2,2} & \cdots & r_{2,m} \\ \vdots & \vdots & \ddots & \vdots \\ r_{m,1} & r_{m,2} & \cdots & r_{m,m} \\ \end{bmatrix}$$ Math representation cited from [1].


Question:

To calculate the Coefficient of Multiple Correlation $R^2$ in my case of a polynomial regression of degree 3 with interaction terms, do I have to take into account the correlation of the input regressors $x_1,\, x_2,\, x_3$ or of all terms of the resulting polynomial $x_1,\, x_2,\, x_3,\, x_1^3,\, x_1^2x_3,\, x_2x_3$?

Only of the input regressors would mean for $c_{y,x}$: $$\boldsymbol{c}_{\mathbf{y},\mathbf{x}} = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}$$ Whereas of all terms would mean: $$\boldsymbol{c}_{\mathbf{y},\mathbf{x}} = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ x_1^3 \\ x_1^2x_3 \\ x_2x_3 \end{bmatrix}$$ Equivalently for $R_{x,x}$.

$\endgroup$
2
$\begingroup$

Question: To calculate the Coefficient of Multiple Correlation $R^2$ in my case of a polynomial regression of degree 3 with interaction terms, do I have to take into account the correlation of the input regressors $x_1,\, x_2,\, x_3$ or of all terms of the resulting polynomial $x_1,\, x_2,\, x_3,\, x_1^3,\, x_1^2x_3,\, x_2x_3$?

The latter - the relevant correlations are for all pairs of vectors in the design matrix. So if you use polynomial regression, you will have vectors of power terms in the design matrix, and the correlations of those vectors with the other vectors would be included in the matrices $\boldsymbol{c}_{\mathbf{y},\mathbf{x}}$ and $\boldsymbol{R}_{\mathbf{x},\mathbf{x}}$.

$\endgroup$
  • $\begingroup$ Thanks alot! So this explicitly means x_1^3 for example will be included, but other combinations dropped out of the solution (c_n = 0) will not be regarded in the matrices. $\endgroup$ – Scotty1- Mar 4 at 11:45
  • 1
    $\begingroup$ Whatever vectors are in the design matrix need to be included in the pairwise correlation matrix. $\endgroup$ – Ben Mar 4 at 12:19
  • $\begingroup$ Thanks again and sorry for the quite stupid follow-up question: Is the mutliplication in $R^2 = c^T R^{-1} c$ a matrix multiplication? So for implementing it in python, is this correct: r_sqrd = np.matmul(np.matmul(c_yx.T, np.linalg.inv(r_xx)), c_yx)? $\endgroup$ – Scotty1- Mar 4 at 16:45
  • $\begingroup$ Some testing suggests that it is a dot-product, yielding a singular value instead of a shape (m, ) vector (which has the same value in all fields). Is that so? $\endgroup$ – Scotty1- Mar 4 at 16:51
  • 1
    $\begingroup$ @Scotty: That formula is a quadratic form, which is a matrix calculation, but leads to a scalar output. The dimensions of the relevant matrix multiplication are $(1 \times m) \times (m \times m) \times (m \times 1) = (1 \times 1)$. $\endgroup$ – Ben Mar 4 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.