ROC curves : using package pROC : DUMMY EXAMPLE

Question

I have been reading a lot about ROC curves, but honestly I still don't have it cleared in my mind. So I request anyone here who could explain me about it with respect to my dummy example below.

So lets say I have a microarray data as follows:

Sites Samp1 Samp2 Samp3 Samp4 Samp5 ... SampN
siteA 0.675 0.344 0.543 0.564 0.098     0.433
siteB 0.345 0.432 0.454 0.122 0.789     0.332
siteC 0.322 0.234 0.987 0.455 0.433     0.765
siteD 0.876 0.455 0.654 0.987 0.332     0.093
siteE 0.543 0.345 0.234 0.123 0.127     0.654

and lets say I have a phenotype for each of the above samples.

Samples Pheno
Samp1 43
Samp2 45
Samp3 56
Samp4 50
Samp5 41
..
SampN 59

And then i run a continuous linear regression for my above data to see which of the sites (may be genes or methylation sites) are highly significant with respect to my phenotype.

So I have my dummy results as follows.

Sites       t-statistics    p-value   Bonferoni Holm   FDR significant
siteD   222.255.348.790.264 6,94E-70    TRUE    TRUE    3,22E-64
siteA   160.991.598.630.311 4,30E-36    TRUE    TRUE    9,96E-31
siteE   154.406.449.263.392 1,01E-32    TRUE    TRUE    1,57E-27
siteB   153.199.072.926.937 4,13E-32    TRUE    TRUE    4,79E-27
siteC   148.394.475.170.859 1,05E-29    TRUE    TRUE    9,77E-25

So till now, I have this. Now I wish to draw an ROC curve for this. I do have an idea that ROC shows the amount of false positives and false negatives. So my questions are :

1) How do I get an ROC for the above ? What kind of input am i supposed to use to get an ROC curve for the above data. How to i get the FP and FN for my "Sites".

2) It would be really helpful if somebody can help me do a small ROC for the above. I am planning to use the package pROC. But I am confused with my inputs there.

Answer 1

You can't have ROC without a grand truth, which is actually pretty hard to obtain in case of feature selection (i.e. site selection in your case). You likely want to make ROC for phenotype prediction, obtained from validating your model on some test set or via cross-validation.

Moreover, p-value of the parameter significance in linear regression is a terrible way of doing feature selection in a large p small n case -- regression will likely overfit and the p-values produced will be a pure noise.