2
$\begingroup$

I have a dataset of 20 observations which correlate solar panel output with meteorological factors and geographical latitude (a total of 3 predictors). When I build a non-linear regression model for this entire dataset, I get an R-squared of 71%. However, when I divide the dataset into two sets of 10, I get an R-squared of 1.

What puzzles me further is that this is true for both of the two datasets I get by dividing by 2, arbitrarily. When I limit my dataset to the 14 data points which lie north of the equator, I get an R-squared of 73%. Obviously I have too few points south of the equator to compare reasonably right now.

What am I doing wrong? Is the model with R-squared = 1 over explained or something?

To reply to a suggestion, I attach below the residuals-vs-fitted

In response to a request

In response to Sextus, here is some output from R:

lm(formula = yields_differences[11:20] ~ poly(latitudesforplotting[11:20], 
    3) + poly(humidity_average_ordered[11:20], 3) + poly(insolation_annual[21:30], 
    3))

Residuals: ALL 10 residuals are 0: no residual degrees of freedom!

Coefficients:
                                          Estimate Std. Error t value Pr(>|t|)
(Intercept)                                  52.55         NA      NA       NA
poly(latitudes[11:20], 3)1      1632.45         NA      NA       NA
poly(latitudes[11:20], 3)2      2949.47         NA      NA       NA
poly(latitudes[11:20], 3)3      2585.41         NA      NA       NA
poly(humidityvalues[11:20], 3)1 -2450.67         NA      NA       NA
poly(humidityvalues[11:20], 3)2  -854.00         NA      NA       NA
poly(humidityvalues[11:20], 3)3 -4182.23         NA      NA       NA
poly(irradiancevalues, 3)1        -3060.18         NA      NA       NA
poly(irradiancevalues, 3)2         -662.06         NA      NA       NA
poly(irradiancevalues, 3)3        -2318.58         NA      NA       NA

Residual standard error: NaN on 0 degrees of freedom
Multiple R-squared:      1,     Adjusted R-squared:    NaN 
F-statistic:   NaN on 9 and 0 DF,  p-value: NA
$\endgroup$
4
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – whuber Jul 30 '20 at 13:43
  • $\begingroup$ Also: stats.stackexchange.com/questions/168599/… $\endgroup$ – kjetil b halvorsen Jul 30 '20 at 14:00
  • $\begingroup$ @kjetilbhalvorsen adjusted $R^2$ will not be computable when the non-adjusted $R^2$ is equal to 1, or more specifically when the degrees of freedom for the error are zero. (it will involve a division like 0/0) $\endgroup$ – Sextus Empiricus Jul 30 '20 at 14:06
  • $\begingroup$ @Sextus Empiricus: I thought about the answer neither of those. Anyhow, it seems these is not nonlinear regression at all ... $\endgroup$ – kjetil b halvorsen Jul 30 '20 at 14:08
2
$\begingroup$

Unless you made some computation error....

When your model fits 10 training data points perfectly but does not predict 10 other testing/validation data points, then you have indeed overfitting.

It is likely that your non-linear model has sufficient flexibility and free parameters to fit any other set of 10 points (even pure noise).


You have 20 data points so you should be less worried about overfitting. You could test this with some sort of cross validation. However, maybe you could first consider whether your model really needs 10 parameters to be fitted (I guess your model has so many parameters based on the idea that it perfectly fits any cut of 10 points).


In response to your edit: Now it is obviously clear that you are fitting/estimating 10 free parameters/coefficients (and lm is not a non-linear model, it is only your predictors which are non linear functions, polynomials, of some input variables). You are estimating 10 parameters (1 intercept and 3x3 coefficients in the 3 polynomials). So that is the reason why you get a perfect fit ($R^2=1$), your problem is over-determined.

$\endgroup$
8
  • 1
    $\begingroup$ Okay, and would it help then to drop some of the independent variables/parameters? I should point out that I am able to get R-squared above 70% with various other models, so I don't think my approach is crazy. $\endgroup$ – Abed Jul 30 '20 at 7:33
  • $\begingroup$ @Abed I would't suggest to drop a variable, but instead less free variables (they can still be in the model, but as fixed). To me a model with such flexibility sounds suspicious, and before considering all kinds of statistical tricks to restrain it, I would consider whether we actually need all that flexibility, and maybe fix or combine some parameters (what's the research question?). I doubt that this exercise (with only 20 points) can be about estimating 10 parameters in a non-linear function of only two variables (and I wouldn't be surprised if that function can be simplified) $\endgroup$ – Sextus Empiricus Jul 30 '20 at 9:36
  • $\begingroup$ Thanks for your help, Sextus. Just to be clear: I do not have 10 parameters, I have three: solar insolation, latitude and humidity. I use these to predict the electric generated by solar PV panels (in fact, it's a bit more complicated than that, but let's go with this for now). My R-squared improves dramatically when I add any one of these three parameters to the model, so for example irradiance alone does not work to determine PV output. Not to start a pattern, but: I would be very happy to chat about this by email for example. $\endgroup$ – Abed Jul 30 '20 at 9:44
  • $\begingroup$ @abed With parameters I mean the unknown coefficients in the model that you are fitting. Not the variables that are the input of the model. $\endgroup$ – Sextus Empiricus Jul 30 '20 at 9:50
  • $\begingroup$ Yes, thank you for clarifying that this is not the same thing as the independent variable. This was not entirely clear to me. Anyway, I have 3 independent variables, 1 dependent variables and a training set composed of up to 20 data points. I need to figure out the best model, and some models are suspiciously good, which is what brought me here today. $\endgroup$ – Abed Jul 30 '20 at 9:57
0
$\begingroup$

Rsquared= 1 indicates that your hypothesis is able to explain the process perfectly which cannot happen and it's a clear sign of overfitting. The reason maybe because your hypothesis is able to capture the trend perfectly by chance since you have very few observations. In short, If you are modelling a random/stochastic process like in your case, you cannot never achieve 100% results.

$\endgroup$
1
  • $\begingroup$ I agree with you. It's clearly a case of overfitting and because of high model complexity to model just 10 observations. However, by chance I meant it maybe capturing the random noise which indeed contributes to the overfitting. Thanks for pointing out though. $\endgroup$ – Vivek Jul 30 '20 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.