1
$\begingroup$

I am trying to create a regression comparing resampled data from 2 sampling periods using $n=6$ samples. I have tried a few different models (e.g., linear, log-transform, and exponential), and have received mixed results.

For example, the linear models look best graphically, but because I have so few data points, each point has a strong effect on the trend and therefore points with greater distance from the mean are resulting in the model having a negative adjusted $R^2$ value.

An exponential model has a much improved, and positive, adjusted $R^2$ value, but graphically doesn't make as much sense as the linear model. Essentially, the exponential model is reacting to the dispersion of points around the linear regression line and trying to make up for the residual error of more outlier-ish points. However, these points would almost certainly be within reasonable distance from the mean if more points were added.

So I have 2 questions:

  1. Do I go with what looks more sensible graphically, or just blindly believe the adjusted $R^2$?
  2. Obviously, more data points would be beneficial, but what are my options if I can't supplement with more data?

Examples:

3 variables demonstrating this issue. Solid color lines are model fits and dashed lines are 95% CI. Black = linear, red = log-trasnformed, green = exponential. x-axis = 1977, y-axis = 2015. Created in R using lm().

Example1 Example2 Example3

Update:

Background info: I have 21 soil variables I resampled in 1977 & 2015. Each of these variables had 6 sample plots shared b/w each sampling period. However, I have a number of 1977 samples & 2015 samples that were only sampled once (in 1 year & not the other). The soil processing was different b/w years, so the 2 years aren't directly comparable. I'm trying to 'correct' the non-resampled 1977 samples to match 2015 data (not sampled in 1977) by regressing my 6 resampled samples for each variable. I'm assuming soil has not changed in 40 years, so I'm not trying to compare soil change, but rather trying to lump all extant samples (resampled and not) for common analyses.

My example data:

dput(soil.dat)
structure(list(plot.2015 = c(5L, 10L, 24L, 25L, 44L, 51L), Silt.2015 = c(28.88, 21.84, 22.89, 25.34, 16.96, 22.36), Sand.2015 = c(63.92, 73.67, 60.38, 69.03, 81.6, 70.18), BS.2015 = c(70, 60.4, 43.2, 41, 84.4, 54), Silt.1977 = c(44L, 38L, 40L, 42L, 50L, 36L), Sand.1977 = c(37L, 49L, 37L, 36L, 39L, 42L), BS.1977 = c(36.6, 26.48, 44.08, 36.6, 53.32, 44.08)), .Names = c("plot.2015", "Silt.2015", "Sand.2015", "BS.2015", "Silt.1977", "Sand.1977", "BS.1977"), class = "data.frame", row.names = c(1L, 2L, 15L, 16L, 18L, 19L))
$\endgroup$
7
  • $\begingroup$ I wouldn't use the R² for model selection. Also for clarification and a better understanding: what's the relation of interest that you want to analyse statistically? Possibly there are better methods than linear regression when you have such a small N. $\endgroup$ Nov 21, 2015 at 18:55
  • $\begingroup$ @BlankUsername: see my update $\endgroup$ Nov 21, 2015 at 19:06
  • $\begingroup$ Can you explain the figures in a few words please? $\endgroup$
    – Michael M
    Nov 21, 2015 at 19:08
  • $\begingroup$ @Michael added some quick notes $\endgroup$ Nov 21, 2015 at 19:12
  • $\begingroup$ It is still unclear what is your data, your models and what is the rationale behind your models. Have you considered Bayesian model? See also: stats.stackexchange.com/questions/12900/… $\endgroup$
    – Tim
    Nov 21, 2015 at 19:21

1 Answer 1

4
$\begingroup$

The problem is that your data is both very small and very noisy. This leads to the very different outcomes you get from your models. In such small samples outliers have very big influence on your outcome. Imagine, for example, that you removed case 44 from the first plot - then we would see "obvious" upward, quadratic trend. Now imagine we removed case 5 - the "trend" would be very different. If we removed case 10 from the second plot we would get very steep positive trend. All this makes the approach you would choose for analyzing this data have great influence on the outcome.

One approach you could possibly apply is to use Bayesian model with informative prior based on your out-of-data knowledge about the phenomenon you observe.

With such a small sample overfitting is a significant risk, so simpler (e.g. linear) models are safer that more complicated ones (e.g. including polynomials).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.