3
$\begingroup$

I'm using data from a proteomics platform called Olink. In this case, the data comes from samples of 16 patients and 16 controls. The patients can be further classified into 2 subgroups. The assay measured 92 proteins in multiplex, and the results were normalized to internal standards and log-transformed. This was all done by other people, I had nothing to do with sample selection or data preprocessing, and this is not my project so I can't change the goals of the analysis.

I am trying to find proteins that are altered between patients and controls or between the 3 subgroups. I know the age of all subjects, so I also wanted to try controlling for age as a continuous covariate. My response variable is protein concentration. I decided to use multiple linear regression and model each protein as a function of one of the classes (patient/control or subgroup1/subgroup2/control) and age. I used the lm() function in R and used the significance of the contribution of each coefficient to determine if any proteins are different between classes and selected those for further analysis.

The formula looks like this: Concentration ~ Class + Age

Where Class represents either patient/control or subgroup1/subgroup2/control.

So once I got the models, I extracted the coefficients from each feature and got their p-value. Then I corrected for multiple comparisons with FDR and selected the proteins that were significant after that.

After much back-and-forth and many iterations of this question, I'm thinking that for this kind of analysis I may not need to do a train/test split or use caret to tune parameters. Maybe I can just use lm() and treat it as using ANOVAs/ANCOVAs? Is that the case, or do I have to do the normal machine learning steps?

$\endgroup$
10
  • 1
    $\begingroup$ I don't understand what you're doing. What does class membership (classification?) have to do with linear regression? What is your research question? What are you trying to achieve? $\endgroup$ Commented Jun 18 at 12:28
  • $\begingroup$ What does this: "selecting the molecules where class is significant" mean? "Significance" is a feature of a statistical test. What are you testing? $\endgroup$
    – Peter Flom
    Commented Jun 18 at 12:31
  • $\begingroup$ @PeterFlom I'm testing whether the coefficient for each feature is different from 0. lm() outputs a t statistic by dividing the estimate by the standard error, and it gives you the corresponding p-value. $\endgroup$ Commented Jun 18 at 13:59
  • $\begingroup$ That's true, but it doesn't seem to have anything to do with my question (or with the question you originally asked). $\endgroup$
    – Peter Flom
    Commented Jun 18 at 15:07
  • 1
    $\begingroup$ @SextusEmpiricus alright, I'm really trying here $\endgroup$ Commented Jul 1 at 11:29

2 Answers 2

3
+50
$\begingroup$

A newly edited version of this question clarifies that the interest is in identifying which among 92 proteins have differential expression between 16 normal controls and 16 patients, with the patients further subdivided into 2 groups. This reasonably common scenario can be handled by well-established methods originally developed decades ago for RNA microarray data but readily adapted to proteomics. A classic set of tools is provided by the limma packge in Bioconductor. A Github resource shows how to proceed with proteomic data.

The overall workflow is similar to the multiple linear modeling with control of false discovery rate (FDR) proposed in the question (and nicely elaborated by Robert Long in another answer), but it takes advantage of the multiple proteins analyzed to get better, pooled estimates of error terms. That makes the analysis less affected by vagaries of individual measurements and can increase power to find true differential expression.

The "normal machine learning steps" of train/test splits aren't suitable unless you have several thousands of cases. See this post by Frank Harrell. For this analysis there don't seem to be any parameters to tune; that's more of an issue if, for example, there is penalization as with principal components or ridge regression, or with LASSO. As noted in that link, bootstrapping is a good way to validate models of this smaller scale (more precisely, to validate the model-building process).

The question will still remain, however, whether any "statistically significant" differences in expression (based on FDR) are of practical significance. For that you have to apply your and your colleagues' understanding of the subject matter.

$\endgroup$
12
  • 2
    $\begingroup$ The probability of replication of the findings in a new sample is close to zero with that number of candidate features and that tiny effective sample size. Sure a paper can be published, but results will be virtually uncorrelated with the truth. $\endgroup$ Commented Jul 1 at 22:16
  • 1
    $\begingroup$ @FrankHarrell Oh, I'm well aware of that. There's just nothing I can do about it. In theory, the idea is to get all the patients with this disease in the country (it's a very very rare condition) into the study, but I'm not holding my breath. This is really so I can exhaust all possible ways to analyze the thing so they'll believe me when I tell them there's very little to report here. $\endgroup$ Commented Jul 2 at 8:27
  • $\begingroup$ @EdM I did use the limma package at one point, and I got the same results as the t-tests. $\endgroup$ Commented Jul 2 at 8:28
  • 1
    $\begingroup$ @FrankHarrell How should the results differ for me not to worry? Also, I don't want to publish garbage either, but it's out of my hands. Or is there a way to prove that those results are incorrect? $\endgroup$ Commented Jul 2 at 12:48
  • 1
    $\begingroup$ One purpose of data reduction is to avoid arbitrary feature selection, so no. There's not enough information in the data to know something's redundant with regard to prediction, but you can do redundancy analysis on predictors alone. Several things are illustrated in this case study: hbiostat.org/rmsc/bacteremia $\endgroup$ Commented Jul 3 at 19:50
2
$\begingroup$

If I understood you research question, it seems like you’re navigating a problem involving identifying molecules that are altered between two categorical classes while controlling for a continuous covariate.

1. Clarify Your Research Question:

  • My understanding of your research question is that you you want to determine if the concentration of each molecule differs between two categorical classes (eg, treatment vs. control) while accounting for the effect of a continuous covariate (eg, age, weight, etc).

  • Your goal is to find molecules where the categorical class has a significant effect on the concentration, after controlling for the covariate.

2. Model Selection:

  • For each molecule, fit a linear regression model with the concentration as the response variable, and the categorical class and the continuous covariate as predictors.
  • The formula for each model would be: Concentration ~ Class + Covariate.

3. Implementation in R:

  • Ensure your data is in a suitable format, with molecules as columns and samples as rows, and include the class and covariate information:
# Example data format
data <- data.frame(
  Class = factor(c('A', 'B', 'A', 'B')),
  Covariate = c(23, 45, 21, 50),
  Molecule1 = c(1.2, 3.4, 2.1, 4.3),
  Molecule2 = c(2.2, 3.5, 1.9, 4.0)
  # Add more molecules as needed
)

4. Running Multiple Models:

Loop through each molecule, fit the linear model, and store the results:

results <- list()
for (molecule in colnames(data)[-c(1, 2)]) {  # Assuming first two columns are Class and Covariate
  model <- lm(as.formula(paste(molecule, "~ Class + Covariate")), data = data)
  results[[molecule]] <- summary(model)$coefficients
}

5. Extract Significant Results:

  • Extract the p-values for the Class variable from each model and adjust for multiple comparisons using a method like Bonferroni or Benjamini-Hochberg:
p_values <- sapply(results, function(x) x['ClassB', 'Pr(>|t|)'])  # Adjust if Class has different levels
p_adjusted <- p.adjust(p_values, method = "BH")  # Benjamini-Hochberg adjustment

significant_molecules <- names(p_adjusted)[p_adjusted < 0.05]

6. Model Diagnostics: Check the diagnostic plots for each model to ensure the assumptions of linear regression are met (linearity, homoscedasticity, normality of residuals):

for (molecule in significant_molecules) {
  model <- lm(as.formula(paste(molecule, "~ Class + Covariate")), data = data)
  par(mfrow = c(2, 2)) # May need to save the plots to disk in some environments
  plot(model)
}

In case the plots do not display in your environment, we can save them to disk instead:

# Directory to save plots
plot_dir <- "diagnostic_plots"
if (!dir.exists(plot_dir)) {
  dir.create(plot_dir)
}

for (molecule in significant_molecules) {
  model <- lm(as.formula(paste(molecule, "~ Class + Covariate")), data = data)
  
  # Create a file name for the plot
  plot_file <- file.path(plot_dir, paste0(molecule, "_diagnostic.png"))
  
  # Open a png device
  png(filename = plot_file)
  
  # Plot diagnostics
  par(mfrow = c(2, 2))
  plot(model)
  
  # Close the device
  dev.off()
  
  # Optionally print a message indicating the plot was saved
  cat("Diagnostic plot saved for", molecule, "as", plot_file, "\n")
}

7. Validation and Model Quality:

  • It is important to validate your models. Splitting your data into training and testing sets and evaluating the performance can help. For linear models, prediction accuracy can be assessed using metrics like R-squared, RMSE (Root Mean Square Error), etc.:
library(caret)
set.seed(15)
trainIndex <- createDataPartition(data$Class, p = .8, list = FALSE, times = 1)
dataTrain <- data[trainIndex,]
dataTest <- data[-trainIndex,]

# Train models on training set and evaluate on test set
for (molecule in significant_molecules) {
  model <- lm(as.formula(paste(molecule, "~ Class + Covariate")), data = dataTrain)
  predictions <- predict(model, newdata = dataTest)
  actuals <- dataTest[[molecule]]
  print(paste(molecule, "R-squared:", summary(lm(predictions ~ actuals))$r.squared))
}

Additional Considerations:

  • Residual Analysis: Check the residuals for normality, homoscedasticity, and independence.
  • Assumption Checks: Ensure all assumptions of linear regression are tested (eg, linearity, no multicollinearity).

Summary:

  • Clarify Research Question: Identify significant molecules controlling for covariate.
  • Model Selection: Use linear regression models for each molecule.
  • Implementation in R: Structure data and fit models.
  • Extract Significant Results: Adjust p-values and identify significant molecules.
  • Diagnostics: Check diagnostic plots to ensure model assumptions are met.
  • Validation: Split data, train, and test models to ensure robustness.

This approach helps to ensure that your findings are statistically valid and robust, accounting for multiple comparisons and ensuring model quality through diagnostic checks and validation.

$\endgroup$
10
  • 2
    $\begingroup$ The feature selection part of the analysis is doomed. This will be exposed by computing an importance measure (absolute $z$ statistic, likelihood ratio $\chi^2$ statistic, etc.) for each feature and bootstrap the process to get confidence intervals on I the importance measures. Examples are here and here. $\endgroup$ Commented Jun 30 at 11:12
  • $\begingroup$ Sorry a link was incorrect. Use hbiostat.org/rmsc/validate $\endgroup$ Commented Jun 30 at 11:17
  • $\begingroup$ @FrankHarrell What will the importance measure show? You mean that the CIs will include 0 for all the models? $\endgroup$ Commented Jul 1 at 8:54
  • $\begingroup$ Importance = 0 is not an interesting value. You need to estimate the amount of importance, and hypothesis testing against zero is very boring. More to the original point, importance measures are a basis for selecting “winning” features if you must, and getting uncertainty intervals for them shows the difficulty (and usually the impossibility) of doing reliable feature selection. $\endgroup$ Commented Jul 1 at 11:44
  • 1
    $\begingroup$ Example: of the 92 proteins suppose that many are co-expressed so that there are 7 clusters of interrelated proteins, with very low correlation between a protein in one cluster and a protein in any other cluster. Score each cluster with the first principal component. Use these component scores to predict Y. A related but sometimes better alternative: sparse principal components analysis. All of this is discussed in RMS. $\endgroup$ Commented Jul 2 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.