am currently working on a problem where I have to calibrate weather parameters at a ground location using Satellite data available (1979-2012) over rectangular grid points and surface observatory data for a location (1980-2000). I have built two models - One for January months over 1980-2000 period and other taking data from November to April over 1980-2000.

Now the problem arises when I compare adjusted R-squared for both the models. Nov-Apr model gives me higher R-squared (0.7 vs. 0.2) whereas residual errors tell me the opposite story, giving poor results in the case of Nov-Apr model.

One option would be to calculate different components of R-squared and see if it can explain the differences. So how do I calculate and explain these components in R? or is there any better and efficient way?

Thanks in advance.

  • $\begingroup$ I can guess what is going on, but you need to give more information. The dependent variable is (daily?) surface temp and the independent variable is (daily?) lower troposphere temp for a "rectangle" including the surface station? Is that true? $\endgroup$
    – Bill
    Oct 23 '13 at 14:18
  • $\begingroup$ @Bill Dependent variable - Tmax daily (Surface Observatory Data). For independent variables, we have daily data for four time periods(5:30, 11:30, 17:30, 23:30). So for Tmax daily we are using reanalysed data from nearest grid point(only) at time 11:30 & 17:30 as independent variables - Tmax, RH, solar radiation (Long & Short Wave), wind (N-S & E-W). So total of 11(=1+2+4+4) independent variables. Do let me know if that is only you meant to confirm. $\endgroup$
    – Mudit
    Oct 25 '13 at 10:14
  • $\begingroup$ OK, I hope to be able to get back to you today. If not, then Monday. $\endgroup$
    – Bill
    Oct 25 '13 at 14:02
  • $\begingroup$ @Bill, Hope you are fine. Could you share your views on problem? $\endgroup$
    – Mudit
    Oct 30 '13 at 4:44

Sorry about the delay in answering. What's going on, I think, is seasonality.

Suppose there are three temperatures. First there is "true" temperature---whatever average high surface air temperature actually occurs on each day. Second, there is ground measured temperature. Third, there is satellite measured temperature. The second and third temperatures are the first temperature, except measured with error. Like this: \begin{align} \textrm{ground}_{d,y} &= \textrm{true}_{d,y} + \nu_{d,y}\\ \textrm{satellite}_{d,y} &= \textrm{true}_{d,y} + \eta_{d,y}\\ \end{align}

Above, $\mathrm{true}_{d,y}$ means the true temperature in year $y$ on day $d$, where day counts up from 1 on 01/01/year to 365 on 12/31/year --- I am too lazy to think about leap years here.

Now, if you regress ground on satellite, of course you are going to get a regression with some predictive power, because both measurements are being driven by the same true temperature. However, the predictive power will not be perfect. How good will the predictions be? Well, that depends on the ratio of signal (variance of movements in true temperature) to noise (variance of movements in $\nu$ and $\eta$. In fact, you can calculate the covariance and correlation between ground and satellite (assuming that the $\nu$ and $\eta$ are iid, independent of each other, and independent of true): \begin{align} Cov(\mathrm{ground},\mathrm{satellite}) &= V(\mathrm{true})\\ Corr(\mathrm{ground},\mathrm{satellite}) &= \frac{V(\mathrm{true})} {\sqrt{(V(\mathrm{true})+V(\nu))(V(\mathrm{true})+V(\eta))}} \end{align}

Now, in the simple bivariate model, when you just regress ground on satellite, $R^2$ is equal to the correlation above squared, or: \begin{align} R^2 &= \frac{(V(\mathrm{true}))^2} {(V(\mathrm{true})+V(\nu))(V(\mathrm{true})+V(\eta))} \end{align}

So, why is $R^2$ so much higher in the Nov to Apr sample than in the Jan sample? That's easy. As long as you are looking at a weather station far from the equator, there are big month-to-month differences in true. So, the more of the year you include in your time window, the higher is $V(\mathrm{true})$, and the higher is $R^2$.

Why, then, is the mean of the sum of squared residuals higher in the Nov to Apr sample than in the Jan sample? The mean of the sum of squared residuals goes in probability to: \begin{align} \frac{1}{n}\sum e_i^2 &\rightarrow V(\mathrm{ground}) + \beta^2 V(\mathrm{satellite}) \end{align}

The $\beta$ in the above is the probability limit of the regression coefficient. Notice that this quantity goes up as the variance of both ground and satellite go up. As you expand your time window from Jan only to Nov through Apr, the variance of true goes up and this drives the variance of both ground and satellite up. In turn, this drives the variance of the residuals up.

If you prefer a numerical example, here is some R code where you can see what is going on. All I have done is implement the reasoning above in a very simple monte carlo. As you can see if you run the code, I have pretty well replicated your $R^2$ and mean squared residual results using somewhat realistic temperature data:

# This program written in response to a Cross Validated question
# http://stats.stackexchange.com/questions/73544/multiple-linear-regression-models-comparison-based-on-r-squared-and-residual-err#comment143511_73544

# The program is a toy monte carlo.
# It generates a "true" but unobservable-to-the-analyst high temperature series
# in degrees Celcius for a location in the northern hemisphere of the earth 
# pretty far from the equator.  It is very loosely based on Des Moines, IA.
# The series simulates seasonality with a sinusoidal curve and generates 
# temperatures for all days from 1980-2000, ignoring leap years.

# Then it generates two series which measure the true temperature with error,
# calling them satellite and ground.  Then it regresses ground on satellite 
# by OLS over various slices of the data.  The slices are only January,
# only Nov through Apr, and all year long.

# day is days since 12/31/1979, ie it starts at 1 on 01/01/1980
# I ignore leap years and assume 365 day years
# day maxes out at 20*365=7300

day <- seq(1:7300)

true.temp <- 32*sin(((day-1)/365)*2*pi + 3*pi/2) + 21 + 4*rnorm(7300)


# Two fallible measures of true temp

satellite <- true.temp + 4*rnorm(7300)
ground <- true.temp + 4*rnorm(7300)


# Now predict ground with satellite over the whole year
pred.reg <- lm(ground~satellite)

# Now predict ground with sattelite over January
january <- ((day-1) %% 365) %/% 31 == 0
plot(satellite[january],ground[january],xlab="satellite",ylab="ground",main="January Only")
pred.reg <- lm(ground[january]~satellite[january])

# Now predict ground with sattelite over Nov through April
novtoapr <- ((day-1)%%365)>304 | ((day-1)%%365)<92
plot(satellite[novtoapr],ground[novtoapr],xlab="satellite",ylab="ground",main="Nov to Apr")
pred.reg <- lm(ground[novtoapr]~satellite[novtoapr])

Look at the plots of ground against satellite for Jan only and Nov to Apr. You can see the higher $R^2$ in the Nov to Apr:

enter image description here enter image description here

Obviously, in your application there is a lot more going on since you have other right-hand-side variables in your analysis. Also, the measurement errors are probably not nice, normally distributed, iid things like in the monte carlo. Nevertheless, I think what is going on in your data is likely to be what is going on in this monte carlo. In conclusion, I don't think your results are anything to worry about. They are exactly what we would expect in this circumstance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.