I have previously only fit GLM models to small/medium sized data (up to several thousand points, maybe tens of thousands). I always try to be meticulous about checking that GLM assumptions hold where appropriate. These assumptions include but are not limited to:

  1. Normality of error (Q-Q plot, Jarque-Bera, Kolmorogov-Smirnov)
  2. Existence of high leverage outliers (visual inspection, Cook's distance)
  3. Multi-collinearity (correlation matrix, variance inflation factor)
  4. Heteroskedasticity (visual inspection, Breusch-Pagan)

I am currently speccing out an interface to construct arbitrary GLM models using Spark (specifically mllib). This platform will potentially be fitted on much larger data sets than I am used to when using GLMs (millions and millions). I don't have a good intuition around the extent to which each one of these assumptions becomes more or less important as the amount of data gets larger. This is important to know, because the methods for checking these assumptions (especially the ones that involve visual inspection), don't necessarily scale very well.

I'm planning on doing some simulations to see for myself, but thought I would seek expert opinion to make up for my lack of experience in this area.

  • $\begingroup$ Why would "Normality of error" be an assumption of GLMs? Indeed, items 2 and 3 don't seem to relate directly to assumptions as such either. Note that with such large samples, performing hypothesis testing of assumptions is almost certain to lead to rejection, even when the deviation from holding is so tiny as to make no discernable difference to the inference. I think an approach that relies on formal testing of assumptions to scale even less well than visual inspection. ... ctd $\endgroup$
    – Glen_b
    Mar 3 '16 at 0:58
  • 1
    $\begingroup$ I think the main assumptions to worry about with GLMs are 1. linearity of the relationship of $\eta$ with $x$ and the form of the link function, 2. the variance specification implied by the specific family choice, and 3. independence. The last one is hard to check unless one has a specific form of dependence in mind. $\endgroup$
    – Glen_b
    Mar 3 '16 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.