Building a regression model for non-linear relationship

Question

I'm trying to build a regression model that best explains the variable C by a function of A and B. Below is the descriptive statistics of the dataset. There are total of 300 instances in this data.

  A                 B                 C            
 Min.   :-8.8600   Min.   :-2.8900   Min.   :-10000.00  
 1st Qu.:-1.9400   1st Qu.:-0.6062   1st Qu.:   -23.38  
 Median : 0.4685   Median : 1.9350   Median :    -3.09  
 Mean   : 2.3170   Mean   : 2.0131   Mean   :   -41.56  
 3rd Qu.: 7.0425   3rd Qu.: 4.6975   3rd Qu.:    16.30  
 Max.   :20.6000   Max.   : 7.0000   Max.   :   202.00  

From the summary descriptives, and from boxplots and scatterplots, I identified an outlier in the column C. I proceeded to remove the outlier and got the below summary.

       A                B                C                     
 Min.   :-8.860   Min.   :-2.890   Min.   :-315.000  
 1st Qu.:-1.950   1st Qu.:-0.587   1st Qu.: -23.100 
 Median : 0.447   Median : 1.940   Median :  -2.940 
 Mean   : 2.291   Mean   : 2.027   Mean   :  -8.259 
 3rd Qu.: 7.025   3rd Qu.: 4.705   3rd Qu.:  16.300  
 Max.   :20.600   Max.   : 7.000   Max.   : 202.000 

Below are the scatterplots between A and C, and B and C from the dataset without the outlier.

You can see from the plot that B has somewhat of a linear relationship with C, but A does not. I tried fitting polynomial models with A^2 and A^3, but the adjusted R-square does not increase noticeably, with it only being somewhere around 0.4.

In such case, what approaches might I take? I've attached the result of some of the basic regression models that I have fitted below for reference.

Call:
lm(formula = C ~ A + B, data = dat2)

Residuals:
    Min      1Q  Median      3Q     Max 
-229.41  -26.36    1.93   32.99  166.43 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  25.7320     4.2640   6.035 4.75e-09 ***
A            -1.3667     0.5776  -2.366   0.0186 *  
B           -15.2219     1.0974 -13.871  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 55.71 on 296 degrees of freedom
Multiple R-squared:  0.394, Adjusted R-squared:  0.3899 
F-statistic: 96.23 on 2 and 296 DF,  p-value: < 2.2e-16

Call:
lm(formula = C ~ A2 + A + B, data = dat2)

Residuals:
    Min      1Q  Median      3Q     Max 
-221.44  -22.79   -2.73   33.87  187.43 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)  32.31815    4.69006   6.891 3.36e-11 ***
A2           -0.30079    0.09525  -3.158  0.00175 ** 
A             0.96274    0.93170   1.033  0.30230    
B           -15.54204    1.08585 -14.313  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 54.88 on 295 degrees of freedom
Multiple R-squared:  0.4138,    Adjusted R-squared:  0.4079 
F-statistic: 69.42 on 3 and 295 DF,  p-value: < 2.2e-16

Call:
lm(formula = C ~ A3 + A2 + A + B, data = dat2)

Residuals:
     Min       1Q   Median       3Q      Max 
-216.089  -21.384   -2.757   32.428  182.474 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)  36.97146    5.25297   7.038 1.38e-11 ***
A3            0.02190    0.01133   1.932  0.05428 .  
A2           -0.59956    0.18137  -3.306  0.00106 ** 
A             0.75143    0.93383   0.805  0.42166    
B           -15.73565    1.08549 -14.496  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 54.63 on 294 degrees of freedom
Multiple R-squared:  0.4212,    Adjusted R-squared:  0.4133 
F-statistic: 53.48 on 4 and 294 DF,  p-value: < 2.2e-16
```

Answer 1

From the two plots you've provided we see that, marginally, as the absolute value of $A$ or $B$ increases, the variance of $C$ increases dramatically, but the expected value of $C$ does not change much, linearly with the regressors or otherwise.

As a result, if there is no interaction between the effects of $A$ and $B$, you will have to accept that regressors $A$ and $B$ do not explain the mean of $C$ very well.

However, it is possible that the interaction of $A$ and $B$ could have some explanatory value. Perhaps when $B < 0$ and $A > 0$, the mean of $Y$ is large whereas when $B > 0$ and $A > 0$, the mean of $Y$ is small.

To test this, use $A*B$ insead of $A + B$ in your R formula.

Incidentally, whatever happens, you should make sure your residuals are iid before trusting any inference on either your original model or this new model with interactions. This is an assumption of linear regression hypothesis tests, and it appears as though it may be violated in your data.