# Discerning differences in groups with millions of datapoints: is a GLM even valid?

I have a MASSIVE dataset of 22 million shrubs from a basin in the southwest US.

I have selected 2 response variables which are both positive continuous variables: Shrub Canopy Volume (cubic meters) & Distance between closest adjacent shrub center (meters.

I have selected two categorical predictors:

surface type (5 levels: 'Fan Remnants','Fan Piedmont','Alluvial Fan', 'Fan Skirts','Alluvial Flat')

and aspect (direction) 16 levels: "W","N","NNE","NE","ENE","E","ESE", "SE", "SSE","S","SSW","SW","WSW","WNW","NW","NNW")

From these graphs you can see there is a definite relationship between aspect and shrub volume

and a minor one between surfaces

Note all graphs here currently have an exponential fit.

I am trying to compare the groups in a physically meaningful way. my question is: what does that look like statistically/mathematically? Should I be reporting purely parameters and descriptive statistics or should I be trying to compare groups with a GLM?

on the face as well these seem to be distributed according to a Pareto distribution (I have also tried to fit Log-Normal, Exponential, and Power Law). It seems to me like Pareto is a good fit. I did my data fitting in python using SciPy. I am trying to use a GLM to back this fit up. I have a continuous response variable (volume of shrubs) and two categorical predictors (5 Surface Types and 16 Directions (Aspect)).

My GLM responses have been super wacky. I have tried many different families but seem to be getting negative coefficients.

some help on how specifically I can run a GLM with a Pareto Distribution for my data, or even if that is the best step forward. Note that it is very difficult to plot the data in R. So pythonic data exploration is ideal, but I would like to use R for actual analysis. Thanks.

• Such a large sample size is going to result in hypothesis testing sensing practically insignificant deviations as statistically significant. Why would you think of graphical examination?
– Dave
Commented May 26, 2020 at 19:16
• @Dave thanks a lot for the response: I opted for a GLM as it was communicated to me this was more robust than ANOVA for large sample sizes. Can you expand on what you mean by graphics examination? cheers JG Commented May 26, 2020 at 19:20
• Graphical examination...I’ve corrected my typo.
– Dave
Commented May 26, 2020 at 19:22
• I think you have (at least) two issues here. The most important is that it makes no sense to estimate the parameters of a probability distribution using a regression on histogram frequencies. (Well, maybe in an extreme case to obtain starting values for an iterative procedure.) The second is that despite or because of the large sample size you don't have random independent samples from a probability distribution. You need to state what you're going to do with the parameter estimates. Compare the parameters for Fan Skirts with those of Alluvial flats?
– JimB
Commented May 26, 2020 at 19:24
• @Dave thanks, still not sure what you mean: do you mean plotting the data? I have done this, but any scatter plot is difficult to read and computationally expensive on my machine. Commented May 26, 2020 at 19:25

This is just an extended comment as I'm still not clear on the objective (which is likely a problem with me and not necessarily the OP).

While you state that "From these graphs you can see there is a definite relationship between aspect and shrub volume", I'm not seeing the specific relationship you have in mind in part because the first figure has the subplots on all different scales. (So does the second figure but not as much.) However, if one takes the parameter values from the first figure and plots those one does see a definite relationship:

• Totally right about the scales, yeesh! BTW. This plot is awesome and really helpful. My question is. Are those Lambda parameters defensible? Is the fit good enough on the graph that I can report these with confidence? Commented May 26, 2020 at 20:39
• For an exponential distribution these parameter values are essentially the reciprocal of the mean. So a similar plot with the mean will also show the same sinusoidal relationship. Probably the mean rather than the exponential distribution parameter would be easier to interpret for most folks.
– JimB
Commented May 26, 2020 at 21:02
• Cheers, Jim. You have been extraordinarily helpful. Commented May 26, 2020 at 21:24
• Actually had a thought. Would reporting means be better than reporting shrub volume medians wholesale? It seems like an extra step Commented May 26, 2020 at 21:35
• If these distributions have approximately exponential distributions, then the median is just about 0.7 times the mean. So a median wouldn't be adding much of anything. You probably should sit down (virtually) with a statistician.
– JimB
Commented May 26, 2020 at 21:46