1
$\begingroup$

I have the problem that in my binomial-glmm model with a nested random effects structure (30,000 rows located 30 groups nested in 25 groups) the diagnostic plots look okay (as far as I can tell), residuals do not show heavy patterns and some predictors are even significant.

However, the $R^2$ is 0.055. My goal is not to predict new values but only to understand the relationships in the present dataset. Is $R^2$ still important?

Or is it possible that my structure is nonsense and thus leads to the low $R^2$?

Additional info:

there are 25 Wind energy stations (WEA) beeing build. During building these, noise levels are believed to have an impact on porpoises (dolphin like species). There 30 acoustic stations measuring porpoise noises (they can differentiate between ships and fish), temperature value and noise levels (SEL). the WEA sometimes use a noise-muting-device. The aim is to find out whether this device leads to more porpoise clickx during a certain time period. Measuremnts of the stations were done over a year, every minute. however, of interest is only the 100 hours interval after the WEA-build was finished. This is called the HRW-phase (hour related work). There is always one WEA-build per time, not all together. So WEA can be thought of as an event.

Dependent variable:

  • ppm (counted porpoise clicks per minute during an hour) so hourly values
    • zero-inflated count data

Independent variable:

  • SEL a measure of noise, averaged per day (continous)
  • average daily temperature (continous)
  • total building time (continuos = minutes)
  • locations x/y of the stations (still in geographic system, i.e. 54,333 9,3333)
  • hrw (time phase after work, from 0,1,2,...,100. stations do not always measure the full lentgh ,sometimes only 17 hours)
  • noise muting device (binary factor)
  • month as a factor with 7 level (cat.)
  • WEA ID (cat, random effect)
  • acoustic station (cat, random effect)
  • podid at one station, sometimes pod-acoutstic measure systems per station were exchanged (cat)
  • potentially also 4 distance classes

I build the random effects with (hrw|WEA/station) although I am still not sure if this is the right design. the thing is that WEA is an event (there 25) at during an event 30 stations measure ppm. is this then crossed or nested? i am not sure.

although I expected temporal and spatial autocorrelation I did not find any hints on that.

I hope this clarifies the story a bit more.

$\endgroup$
  • $\begingroup$ If you have a binomial-GLiMM model, the traditional $R^2$ does not apply. Are you referring to one of the so-called *pseudo-$R^2$s? $\endgroup$ – gung - Reinstate Monica Sep 1 '12 at 19:25
  • $\begingroup$ So you're basically doing a logistic ANOVA? Also if you have 30 units nested within 25 units, there is going to be identifiability issues as most "level 3" units only have one "level 2" unit nested within them. $\endgroup$ – probabilityislogic Sep 2 '12 at 1:14
  • $\begingroup$ in fact, yes and no. Yes, since all of the IVs are not measured on an hourly basis and some are thus rather "semi continous" if you understand. I was thinking of aggregating the data, but did not find out what would be the best strategy. No because I struggle with simply splitting into 0 and 1 by saying ifelse(ppm ==0, 0,1) since very low values of ppm are possible (range from 1 to 60, of course + 0) $\endgroup$ – Jens Sep 2 '12 at 6:52
4
$\begingroup$

Assuming you are using a pseudo-$R^2$, as @gung commented, a low value of this means that your model is not explaining much (although you should note that the interpretation of pseudo $R^2$ is not identical to that of $R^2$ in OLS models). You could also look at other effect size measures, such as odds ratios.

The fact that some things are significant even with such small effect sizes is due to your large sample. With 30,000 subjects, nearly any effect will be statistically significant. Whether it is of practical importance is another matter. In most fields, such small effects will not be of practical importance.

If you provide context (e.g. what is your DV and IVs?) then someone may be able to provide more help.

$\endgroup$
  • $\begingroup$ Dear Peter, thanks for your answer. At least, also some things were not significant :) Can you quickly explain DV and IV? I did not hear these abbreviations before. I will try to provide more context though. $\endgroup$ – Jens Sep 1 '12 at 20:49
  • $\begingroup$ DV = dependent variable IV = independent variable $\endgroup$ – Peter Flom - Reinstate Monica Sep 1 '12 at 20:55
  • $\begingroup$ ah okay, Info added above! $\endgroup$ – Jens Sep 2 '12 at 6:31
  • 1
    $\begingroup$ @Shaniqueia Finance is one area I know little about. But I'd guess very small improvements in performance can mean a lot in some areas. $\endgroup$ – Peter Flom - Reinstate Monica Sep 2 '12 at 11:35
  • 1
    $\begingroup$ @Shaniqueia Don't know to which papers you are refering to, but it's no rare to find very low $R^2$ when dealing with financial returns in high (daily) and ultra-high(intraday) frequency data. The majority of the models being published tend to refer to issues other than predictions... in many markets this too difficult (that's why $R^2$ are so low and often naive models like random walks perform well) so they adress other issues related to higher moments of financial returns and not necesarily to their expectation (= first moment ->prediction) $\endgroup$ – JDav Sep 2 '12 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.