1
$\begingroup$

I have a neural network that gives two variables X1 and X2 for each data point. For each data point, there is also a target variable Y which I'm interested in predicting (to be more precise, show statistical significance) with X1 and X2.

X1 and X2 are learned from a large dataset and thus large in amount. But the samples for Y is very scarce. Also, Y can have only around 5 possible values for each observation, and thus is extremely noisy, and the correlation of Y with X1 and/or X2 is not so apparent.

When I do linear regression with X1 and Y (not X2), it gives an R-squared of only 6.8%. But, the P-test for X1 gives a value of 0.000 so I think it could be said that X1 is statistically significant. I'm using the OLS model of statsmodel library from python here, with constant term.

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.068
Model:                            OLS   Adj. R-squared:                  0.067
Method:                 Least Squares   F-statistic:                     50.19
Date:                Wed, 21 Apr 2021   Prob (F-statistic):           3.49e-12
Time:                        14:14:54   Log-Likelihood:                -774.96
No. Observations:                 685   AIC:                             1554.
Df Residuals:                     683   BIC:                             1563.
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const         -1.2663      0.086    -14.752      0.000      -1.435      -1.098
x1             0.7582      0.107      7.084      0.000       0.548       0.968
==============================================================================
Omnibus:                       71.552   Durbin-Watson:                   1.990
Prob(Omnibus):                  0.000   Jarque-Bera (JB):               92.665
Skew:                          -0.897   Prob(JB):                     7.55e-21
Kurtosis:                       3.173   Cond. No.                         5.96
==============================================================================

The goal for me is to show that the inclusion of factor X2 adds more to the predictability of Y, or more precisely, that X2 has a statistical significance to the prediction of Y that is independent to that of X1. Running the OLS model with X1 and X2 gives an increase in R-squared to 7.5%. P-test gives a value of 0.026 for X2, showing that X2 is statistically significant.

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.075
Model:                            OLS   Adj. R-squared:                  0.072
Method:                 Least Squares   F-statistic:                     27.73
Date:                Wed, 21 Apr 2021   Prob (F-statistic):           2.65e-12
Time:                        13:51:51   Log-Likelihood:                -772.47
No. Observations:                 685   AIC:                             1551.
Df Residuals:                     682   BIC:                             1565.
Df Model:                           2                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const         -0.9651      0.160     -6.036      0.000      -1.279      -0.651
x1             0.5900      0.131      4.515      0.000       0.333       0.847
x2            -0.4472      0.201     -2.230      0.026      -0.841      -0.054
==============================================================================
Omnibus:                       68.319   Durbin-Watson:                   1.996
Prob(Omnibus):                  0.000   Jarque-Bera (JB):               87.632
Skew:                          -0.874   Prob(JB):                     9.35e-20
Kurtosis:                       3.118   Cond. No.                         12.3
==============================================================================

This is where I'm stuck. I want to show that (i) X1 and X2 solely from the largely available data does sufficient job in predicting Y, and more importantly for me, (ii) X2 does contribute to the prediction of Y in ways X1 can't do.

For (ii) the P-test on X2 is not meaningful. For example, if X2 was exactly equal to X1, the p-value for X2 would also have been equal to 0.000, thus X2 is significant but does not do something new aside from X1. Still, an increase in R-squared by 0.7% also doesn't seem sufficient as well for me. An ideal case for me here is when X1 and X2 are independent, and Y = a X1 + b X2 for some constants a and b, so that each contribute to Y independently. Would there be any way to show how close the situation is to the 'ideal case' I just described?

I apologize if the question is confusing to understand or I am asking wrong questions. I come from a CS background with not much experience in statistics, so I am not understanding exactly what I'm asking and might have an unorthodox way of describing the problem. It would be great if you could shape the question in more precise form if you could.

$\endgroup$
4
$\begingroup$

The "statistical" way of looking at this is Analysis of variance, also known as ANOVA. Essentially, you have two models for $Y$: model 1 explains $Y$ through $X_1$ alone, model 2 explains $Y$ through both $X_1$ and $X_2$. Since the models are nested, model 2's residuals will at most be as large as model 1's, and almost certainly will be smaller. Under certain assumptions, the reduction in mean squared error follows a known distribution, the $F$ distribution. You can use an ANOVA table to assess whether model 2 explains significantly more variance than model 1. You can use the ANOVA functionality in statsmodels for OLS, and if you have access to your network's fits, you can calculate the requisite sums of squares by hand.

However, you write that you are thinking in terms of "predicting". Then I would actually not use ANOVA, which exclusively works in terms of in-sample fitting, not predicting. For the latter use case, you could split your data into a training and a testing sample, then fit both models, and use each model to predict into the testing sample. Then calculate squared errors, and perform a Diebold-Mariano test for superior predictive power of model 2 over model 1.

In any case, note that you probably have a lot of data, so even small differences in fit/prediction accuracy will be statistically significant, whether they are clinically/businesswise significant or not. Thus, also take a look at the effect sizes, i.e., the reductions in mean squared errors.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer! I'll definitely have a look at ANOVA. I would say I'm more inclined to verifying significance than prediction, but I'll try thinking in both terms. $\endgroup$ Apr 21 '21 at 15:10
  • $\begingroup$ Regarding the scale of data, actually X1 and X2 are a lot and learned from a large dataset. The thing is that the target Y is very scarce. I wanted to show that (i) X1 and X2 solely from the largely available data does sufficient job in predicting Y, and more importantly (ii) X2 does contribute to the prediction of Y in ways X1 can't do. I changed the question to reflect this. $\endgroup$ Apr 21 '21 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.