Best statistical analysis with (very) limited samples : MLR vs GLM vs GAM vs something else?

Question

I have a very limited dataset (11 observations). I am trying to obtain relations between a response variable and environmental covariates (I have a set of 14 environmental covariates that I can use). The end goal is to be able to predict the response variable with a combination of the environmental covariates.

I have ran MLRs which give very interesting results, but I'm not sure if they are statistically robust enough, since my data is not-parametric.

I have also ran GLMs and GAMs and also obtained interesting relations.

I know my data is very limited, but it is often the case with this kind of data since each point is very hard to get (time and money expensive).

Answer 1

The end goal is to be able to predict the response variable with a combination of the environmental covariates.

IMO, this little line is key. As Shawn writes, you should not care about which analysis is "interesting" or "gives interesting results". If you want to predict something, you need to look at whether you predict well.

You have very little data. (I don't know what you mean by "my data is not-parametric".) I would very much recommend you include extremely simple methods in your bag of tricks. Things like just taking the overall mean of your data points as a prediction, disregarding all predictors. This is often surprisingly hard to beat, especially in data poor situations (and while that thread and this one are about time series forecasting, the point stands also in your situation).

Your next step should be to use simple linear regression on a single predictor. Do consider averaging the predictions from your 15 simple models (one "overall mean" prediction, and 14 single-covariate predictions); model averaging is usually also surprisingly powerful.

In any case, it will already be very hard to get a feeling for how well your methods will work, because you have so little data that a holdout sample you could base your prediction analysis on will be extremely small. By all means do leave-one-out cross validation, but always keep in the back of your head that you just have 11 LOOCV runs, and that you should treat any conclusions from this tiny sample with a lot of caution.

That said, also think about your covariates. If your goal is to predict your focal variable, do you need to also predict your covariates (and any error in this prediction will immediately carry through to errors in the focal forecast), or are these intervention variables that you can set, so you know their future values with certainty (but you possibly have selection biases and should really think about how the interventions are decided upon)?

Finally, I would suggest you invest a little thinking in what error or accuracy measure you choose to evaluate your models, because there are a few rather surprising pitfalls here (Kolassa, 2020).

Answer 2

First off, your objective here is very backwards. It shouldn't be which method is the most interesting, it should be which method is applicable to my situation. In your situation, a GAM is complete overkill when you don't have to deal with nonlinearity, and is going to be more data hungry then typical techniques. A GLM with a specific residual distribution should really only be applied if you expect the conditional distribution to exhibit a specific pattern (e.g. Bernoulli distribution of a binary response). Linear regression is for modeling a Gaussian conditional distribution and would be applicable in a situation where you don't need to go beyond that assumption.

However, with such a finite amount of data, I can't expect you to get anything of value with $n = 11$ data points. What's worse, you are going to dangerously overfit the model by having almost as many predictors as observations. In fact, such a model should in principle be impossible to fit anyway given you have more predictors than observations, so I think some information may be missing from your question. A very simple demonstration in R with some simulated data shows the results are nonsensical:

#### Setup Parameters ####
set.seed(123) # random seed
n <- 11 # observations 
num_x <- 14 # predictors

#### Simulate X Variables ####
x <- matrix(
  rnorm(n * num_x), 
  nrow = n, 
  ncol = num_x
  )

#### Merge X Data ####
x_df <- as.data.frame(x)
colnames(x_df) <- paste0("x", 1:num_x)

#### Simulate Y and Merge Data Again ####
y <- rnorm(n)
data <- cbind(x_df, y)

#### Fit Regression ###
fit <- lm(
  formula = y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 + x12 + x13 + x14,
  data = data
)

#### Print Model ####
summary(fit)

As shown below (notice it completely drops coefficients at $X_{11}$ and warns us that the matrix has become singular):

Call:
lm(formula = y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + 
    x10 + x11 + x12 + x13 + x14, data = data)

Residuals:
ALL 11 residuals are 0: no residual degrees of freedom!

Coefficients: (4 not defined because of singularities)
             Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.028574        NaN     NaN      NaN
x1           1.148516        NaN     NaN      NaN
x2           0.421618        NaN     NaN      NaN
x3          -0.008065        NaN     NaN      NaN
x4          -0.278369        NaN     NaN      NaN
x5           2.076605        NaN     NaN      NaN
x6          -2.059897        NaN     NaN      NaN
x7           0.661934        NaN     NaN      NaN
x8          -1.367416        NaN     NaN      NaN
x9          -0.074724        NaN     NaN      NaN
x10          0.774342        NaN     NaN      NaN
x11                NA         NA      NA       NA
x12                NA         NA      NA       NA
x13                NA         NA      NA       NA
x14                NA         NA      NA       NA

No matter what technique you use, your results are going to be highly unreliable with such little data anyway. Since you already peaked at this data and threw different combinations of techniques at it, the results are not going to be very trustworthy. I would find a way to either get a new sample as well as find out how to get more data in the future. This will be problematic in your future research if you can't manage get more than eleven observations.

Answer 3

With 14 covariates, I would do one of two things:

• run univariate analyses with the individual covariates as predictors (more or less equivalent to computing the correlation between each covariate and the predictor). Select the largest estimated coefficients/correlations, or the coefficients with the clearest minimum confidence interval bound (i.e. the largest lower bound for positive coefficients/correlations or the most negative upper bound for coefficients estimates/correlations) as possibly worth paying attention to.
• collapse the dimension of your problem by computing the principal components of the predictors; do a univariate regression of the response on PC1.

Unless your data have very low noise, neither of these will be super-reliable in any case ...