I have performed some metabolomic analysis of different tumor sites and would like to see if there are any correlations between the metabolite levels and some of the tumor characteristics that I am interested in. These correlations (if they exist) do not necessary have to be linear relations, so I was trying to understand how to select the best model that would fit my data and to determine if the fit is accurate.

As far as I understand, you can calculate the Akaike Information Criterion (AIC) for each of the individual fits and, in theory, the lower the AIC of a model, the better the model fits the actual data points. My questions are:

  1. Am I correct in my understanding of how AIC works?

  2. Should I be using other parameters in addition/instead of AIC

  3. Lets say I have a dataset that has no true correlation whatsoever. If I attempt to perform non-linear fitting of this data, I will still obtain a value for AIC, that I can compare between models to determine what fits best. However, since there is not really any correlation, this model will be largely worthless. Is there an absolute value of AIC, at which one can simply say, "this model does not fit the data at all?"

As a follow-up to the previous question, if there is a paramater whose value can be used to determine if a model fits the data well, can that parameter/set of parameters be used to calculate a probability of fit, akin to having a p-value calculated for linear correlation? (from what I read, this is much more difficult for non-linear correlations, but I wanted to make sure.

 Let me know if something is unclear.
  • 1
    $\begingroup$ When you say "metabolomic analysis of different tumor sites" do you mean samples taken from the very same tumor at different locations, or do you mean "sites" to be gross anatomical locations like lung, liver, breast, or perhaps subsites within a general anatomic location (e.g., oral cavity, larynx, oropharynx within head and neck)? Also, could please you say a bit more about the nature of the "tumor characteristics" you wish to compare against the metabolomic data? Are these continuous, categorical (ordered or unordered), etc? $\endgroup$ – EdM Apr 29 '20 at 22:15
  • $\begingroup$ The tumor data is biopsies from the same tumor site and the correlating varialbe is, I believe, continuous (it can be any positive number). $\endgroup$ – João Duarte Apr 30 '20 at 8:03
  • $\begingroup$ How many biopsies per tumor, and how many tumors? That will help determine how flexible your non-linear model can be. $\endgroup$ – EdM Apr 30 '20 at 11:25
  • $\begingroup$ At this moment, I have 1 tumor with 14 samples, but will get more datapoints from other tumors in the future. In all, I should get about 5-6 more tumors with similar number of biopsies (14) $\endgroup$ – João Duarte Apr 30 '20 at 16:06

Don't forget that a non-linear relationship between 2 variables can often be modeled with linear regression. The "linear" in linear regression only means linearity in the parameters. Besides obvious choices like prior non-linear transformations of predictor or outcome variables, non-linear relationships can often be modeled flexibly by restricted cubic splines, with parameters estimated in a linear regression model. So there's a good chance that standard linear modeling techniques can accomplish what you want.

One problem however you proceed is that you might not have enough data points to do much more than a single-parameter fit. Another is that even such a regression might be a spurious regression. There are additional issues if you are doing large-scale metabolomics, analyzing hundreds of metabolites, versus just one or a few. So I'll concentrate on those issues first and say a bit about AIC at the end.

Limits on model complexity

To start, consider a simple model with one metabolite and your tumor characteristic, which we'll call "M" and "TC" respectively, modeling M as a function of TC. (This choice is made for concreteness; similar considerations would hold if you model TC as a function of M.) The usual rule of thumb for regression in biomedical research is that you should have 10-20 observations per parameter that you estimate from the data. Otherwise you are in danger of overfitting, finding a relationship that might be true in your particular data sample but that doesn't generalize well to new data.

So your 14 samples from a single tumor are only enough to estimate one parameter, effectively a slope. That could be the slope of a relationship between M and TC in their original scales or in some transformed scales, but even a quadratic model would mean 2 parameters estimated (beyond the mean) so only 7 observations per parameter.

Things might seem to get better with more tumors. With six tumors each biopsied at 14 sites you have 84 observations, in principle enough for estimating 5 or 6 parameter values. But you might gain less than you hope.

Say that there are systematic differences in M values (without corresponding differences in TC values) among the tumors. Then you have a problem. If you ignore those differences there might be too much variability in your parameter estimates to get a useful model. If you take just baseline M values into account as fixed effects with 5 extra tumors then you have to estimate 5 more parameters. A mixed model treating tumors as random factors might help by reducing the number of parameters you have to estimate.

In either fixed or random-effects modeling you would have to consider whether tumors differ mainly in terms of baseline M levels or also in the slopes of the relationship between M and TC. The more systematic differences you take into account among tumors the fewer parameters left for your model of M versus TC. So there will be limits on how complex your non-linear model can be.

Spurious regression

Although spurious regression might be more commonly thought about in time series analysis, it can be similarly important in spatial analysis, which is what you are effectively doing. Say that both M and TC co-vary because of some joint association with another factor that affects both of them. In your case examples could be tumor oxygen level or distance from the invasive front of the tumor (associated with different cancer cell phenotypes, level of immune infiltration, etc). So there will at least have to be great care in interpreting any relationship that you do find, as there might be no direct influence of TC on M or vice-versa.

Multiple metabolites or tumor characteristics

If you have several metabolites $\sf M_i$ and/or tumor characteristics $\sf TC_j$ and do separate models for each relationship then you must address the multiple comparison problem. The more tests you do at some level of statistical significance, the more likely you are to find an apparent "significant" relationship simply by chance. I don't have experience with metabolomics, but in large-scale RNA expression studies the false-discovery rate is usually controlled, accepting that some fraction of the associations are spurious as the cost of not missing some potentially important true associations.

With many metabolites and a single TC you might consider modeling TC against all of the $\sf M_i$ at once, using principal-components or ridge regression or LASSO to get around the problem of many more predictors than observations. With respect to non-linearity, you could still consider some prior non-linear transformation of TC or the $\sf M_i$. For example, RNA expression data are typically log-transformed before such types of analyses.


Your basic understanding of AIC seems sound (your point 1), but AIC might be unnecessary or require extra caution in its use for your application (point 2).

For example, if you fit the original non-linear relationship with restricted cubic splines via a model linear in the parameters, standard analysis of variance of nested models having different numbers of spline knots (i.e., different model complexity) can accomplish what you want. Unlike AIC, analysis of variance will provide estimates of significance in terms of p values, telling you when extra complexity no longer helps.

As you will have relatively small numbers of observations you would probably want to use the correction for small samples, the AICc. That formula only holds strictly, however, for univariate linear models with normal residuals.

If you are comparing non-nested models (e.g., different non-linear transformations of predictors or different choices of predictors) then some think that AIC is inappropriate. This page and its links provide a good introduction to the dispute. Should you be using AIC to compare models with different transformations of the outcome variable then you have to account for the transformations before you do your AIC comparison.

With respect to your point 3, AIC provides no guidance. Some measure of the variance of the outcome observations that is explained by the model provides a guide. Consider that along with what you deem, based on your knowledge of the subject matter, to be practically (as opposed to statistically) significant. If you use linear modeling techniques to describe your originally non-linear relationship between M and TC, then the adjusted $R^2$ provides such a measure, corrected for the sample size and the number of parameters estimated from the data.

  • 1
    $\begingroup$ Just wanted to say that your answer is great and much appreciated. There is much to unpack here and I am still learning a lot about this. I might have some questions in the future, but this certainly helped. Sorry for the delay in responding. $\endgroup$ – João Duarte May 8 '20 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.