Compute R Squared by Fixing Betas for Multi Linear Regression without Intercept results in a large R Square

Question

I want to fix the betas in multi linear regression based on some data I have, which leads to a RSquare value less than 0% and greater than 100 % based on the projection approach mentioned in Tibshirani, Hastie et. all book.

What's the best way to compute RSquare after fixing the beta values for running multi linear regression with no intercept -

Load the Data

import numpy as np
import pandas as pd

import statsmodels.api as sm

data = sm.datasets.get_rdataset('iris').data

Define x and y variables -

x = data.iloc[:, 1:4].values
y = data.iloc[:, 0].values

Solve for betas as per Tibshirani Book -

betas = np.linalg.solve(x.T @ x, x.T @ y)

array([ 1.12106169,  0.92352887, -0.89567583])

sm.OLS(y, x).fit().params

array([ 1.12106169,  0.92352887, -0.89567583])

Alternately, Fixate betas per some understanding of the environment-

alt_betas = [3.7, -10, 45.78]

Now, Compute R Squared in 3 ways -

1. Using Statsmodel with no intercept

2. Using Projection Method for R Sq

3. Using Projection Method but using the Fixated Betas for RSq

Statsmodels

---

sm.OLS(y, x).fit().rsquared * 100

99.61972754365206

Projection

---

(y @ x @ betas / (y @ y) ) * 100

99.61972754365208

Projection with fixed betas

---

(y @ x @ alt_betas / (y @ y) ) * 100

511.1237918393523

Now I understand it should be different given I'm using different betas, but this violates the rule that RSq should be between 0 and 1.

If I had some alternate betas, is there a way to fix it and use statsmodels OLS to compute the R Square?

Think of it as I need to use Alternate Betas as my use case which I think is the true representation of the environment from my perspective.

Thanks in advance!

Answer 1

In OLS linear regression with an intercept, there are multiple equivalent expressions of $R^2$. Four come to mind.

1. Squared Pearson correlation between the feature and the outcome
2. Squared Pearson correlation between the outcome and the predictions
3. Proportion of variance explained
4. Comparison of the square loss incurred by the model to the square loss incurred by an intercept-only baseline model that always predicts the sample mean, $\bar y$, that is: $ R^2=1-\left(\frac{\overbrace{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2}^{\text{Square loss of your model}} }{\underbrace{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2}_{\text{Square loss of the baseline model}} }\right) $.

In this setting, each of these will give the same answer, which will be in the closed interval $[0, 1]$.

If you fit an intercept-free model in R software, it will use a calculation aligned with the philosophy of the four calculation...sort of. The calculation will compare to an intercept-free model instead of an intercept-only model, meaning that the baseline predictions are $0$, no matter the feature values. That is, R will use the following calculation.

$$ R^2=1-\left(\frac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i - 0 \right)^2 }\right)=1-\left(\frac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}y_i^2 }\right) $$

I think this gives a plan to calculate an $R^2$-style calculation for your work. Instead of calculating with the $\hat y_i$ given by the OLS-estimated parameters $\hat\beta$, you calculate the $\hat y_i$ according to the equation with your parameters, $\hat\gamma$.

$$ 1-\left(\frac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\left( \hat\gamma_1 x_{i,1} + \hat\gamma_2 x_{i,2} + \dots + \hat\gamma_p x_{i,p} \right) \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}y_i^2 }\right)\in\left(-\infty, 1\right] $$

Depending on what you want the calculation to tell you, you might want to include the $\bar y$ in the denominator.

$$ 1-\left(\frac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\left( \hat\gamma_1 x_{i,1} + \hat\gamma_2 x_{i,2} + \dots + \hat\gamma_p x_{i,p} \right) \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left(y_i-\bar y\right)^2 }\right)\in\left(-\infty, 1\right] $$

Finally, you might just want to calculate the squared Pearson correlation between between the true values and the predictions coming from using your specified coefficients.

$$ \left[ \text{corr}\left( y_i , \hat\gamma_1 x_{i,1} + \hat\gamma_2 x_{i,2} + \dots + \hat\gamma_p x_{i,p} \right) \right]^2\in\left[0, 1\right] $$

Of the three, this makes the least sense to me, but it might be important for your work, such as a hard requirement to have the value in $[0, 1]$.

It is fine to generalize familiar notions. This is done all over the place in mathematics: I do that here, here, here, here, and here (all of which align with UCLA idea #2 and the Gneiting-Resin $R^*$ in their equation (32)), and I recall the Munkres Topology book giving a description of how the early days of topology required mathematicians to define exactly what they wanted their generalized notion of "open set" to mean. However, you have to have a sense of what you want your generalized notion of $R^2$ to mean if you want to generalize $R^2$ beyond the simple setting.

REFERENCE

Gneiting, Tilmann, and Johannes Resin. "Regression diagnostics meets forecast evaluation: Conditional calibration, reliability diagrams, and coefficient of determination." Electronic Journal of Statistics 17.2 (2023): 3226-3286.