# Loess in gene enrichment analysis using DESeq

Often MA plots (Bland-Altman style representations of gene enrichments) are corrected using the LOESS method. Will a LOESS correction significantly affect the results of gene enrichment analysis? More specifically, will different genes appear as significantly enriched (for instance as defined by the criteria p-value < 10^-3, log2 fold change > 3)?

And is there a recommended implementation to use the LOESS correction together with the DESeq algorithm of Anders and Huber, 2010 (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3218662/)? Ideally, Matlab-based implementations are preferred.

• Are your data count data, as in the cited paper, or are they fluorescence intensity data, as in traditional expression microarrays? My guess is, in any event, you are more likely to find an implementation in R/Bioconductor than in MATLAB. – EdM Aug 27 '15 at 15:25
• Indeed, the data is count-based, measured on Illumina MiSeq. I'm working according to a DESeq-implementation in Matlab according to uk.mathworks.com/help/bioinfo/examples/… . But very much agree with you that I should have started my project in R three years ago! – Simon Aug 27 '15 at 15:27
• Do you think your data need the LOESS correction, based on your MA plots? The MA plot in figure 3 of the cited paper looked quite nice with no need for such correction. – EdM Aug 27 '15 at 15:44
• Yes, in a couple of my MA plots a majority of the data does not lie on the X-axis. As an example, please consider the plots to the right of the diagonal: dropbox.com/s/qssqov0hzr9o9h5/Figure_18_2_v01.png?dl=0 – Simon Aug 27 '15 at 17:00
• As far as I understand the principles of next gen sequencing quantification, this 'bias' of counts towards a condition is a side-effect of genes with extremely high count numbers in the other condition. Ie. such high count outliers 'use up' the counts budget for other, less frequent genes. – Simon Aug 27 '15 at 17:09

To answer your first question, LOESS correction of MA plots was designed to affect the results of gene enrichment analysis. Given the vagaries of 2-color hybridization in early microarray studies, you were likely to overestimate the number of biologically significant changes if you simply analyzed $log_2$ fluorescence intensity differences (M) without taking into account systematic changes in typical M values with overall average expression (A). LOESS corrected for those systematic overall changes along MA plots, in experiments where most genes were not differentially expressed, so that spurious results from these technical limitations were minimized.
So LOESS normalization does not seem compatible with this method. The only normalization this method seems to support is the effective library size. If an equivalent functionality to the DESeq estimateSizeFactors function, which corrects for library size, is available in the MATLAB implementation, make sure you used it. Note that uncorrected different effective library sizes might explain the systematic slopes of the middle portions of your gene-expression clouds away from the x-axis of your MA plots.