We are working with a dataset that has hundreds of biomarkers (many of which are correlated) and often they have many missing values. Our initial goal was to use an elastic net but that would require us to use the complete data and would remove so many observations from the analysis. Hence we decided to use a combo of filtering ($p < 0.10$) and ridge regression. However, there is one more problem. Although the data we have are very unlikely to have huge measurement errors, the outcome versus biomarker $i$ (where $i=1,...,560$) regressions (used for filtering) would identify many outliers. We don't want to remove these so-called 'outliers' identified by the regressions because they probably aren't false values and we don't want to increase the level of missingness. So, we tried standardizing and log transforming the biomarkers (so as to squeeze the range). As expected, the p-values will not change when biomarkers are standardized (and there will be fewer observations identified as 'outliers') but the p-values change a lot when log-transformed (but gives us way fewer observations identified as 'outliers').

My questions: Are these 'regression identified outliers' going to have an adverse effect on the stability of the regressions (and p-values)? Is it recommended that we log transform the regressor to squeeze the range to get rid of the 'outliers' identified by the regressions? If there isn't any problem, as the p-values change a lot, should we still prefer this over standardizing the biomarkers? Do you think we shouldn't try to standardize/log-transform the biomarkers at all to squeeze the range?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.