# Questions about thresholding the data

I came across a data mining course project online.

The data is of samples with 7000 features as genes. Each gene is associated with a value. Some of the values are negative. The data looks like in this way:

SNO "U48730_at" "U58516_at" "U73738_at" "X06956_at" "X16699_at" "X83863_at"

X1 " 27" " 161" " 0" " 34" " 2" " 116"
X2 " 27" " 265" " 0" " 98" " 2" " 123"
X3 " 24" " 126" " 0" " 21" " 0" " 142"
X4 " 27" " 163" " -1" " 16" " -1" " 134"
X5 " 41" " 138" " 1" " 29" " 1" " 153"
X6 " 55" " 107" " -1" " 17" " 0" " 152"
X7 " 27" " 99" " 0" " 57" " 1" " 139"
X8 " 2" " 137" " -1" " 19" " -3" " 213"
X9 " -5" " 161" " -3" " 23" " 2" " 193"
X10 " 0" " 110" " -3" " 7" " -1" " 208"
X11 " -7" " 67" " 1" " 2" " -2" " 149"
X12 " 4" " 93" " 3" " 37" " 2" " 266"
X13 " 2" " 75" " 3" " 30" " 6" " 205"


The professor advise the students to first do 'data cleaning'. The original sentence is Threshold both train and test data to a minimum value of 20, maximum of 16,000.

I first thought that it is to search over each gene and if there is a value out of the bounds, then just discard this gene as a feature. However, it seems for every gene, there must be a sample with the value out of bound.

What should I do by "threshold this data"? Is that like if the value is below 20, then set it 20 or if the value is above 16000, then just set it as 16000?

In fact, I did the last operation in R by

data[data<20] <- 20


and it turns out that the speed of the command is very slow. (79*7070 samples)

• Your latter strategy is winsorising. Statisticians are usually leery of this, & prefer to use robust analyses. As far as what the instructor had wanted students to do, I can't say for sure; your best bet would be to see if you can find it in the materials that are online, or send them an email. Regarding how to speed up these algorithms in R, that's off-topic for CV (see our help page), but should be on-topic on Stack Overflow. Commented Oct 14, 2013 at 20:45
• Not sure - I've not heard of 'thresholding' before. I'd guess it means 'Winsorizing' just because if he'd meant 'discard data less than 20 or more than 16k' it would have been straightforward to say just that. But people don't always like to be straightforward. On the other hand you cross thresholds rather than piling stuff up on them. Commented Oct 14, 2013 at 21:21
• What is the meaning of your data? For example, what is U58516_at? Is it a gene? a specie? an animal? What is X1, X2, etc.? What the positive and negative numbers mean? From the question one can only understand that there are columns starting with U and ending with _at, there are rows starting with X, there are numbers corresponding to each column/row pair, and all this is somehow connected to genetics. Also, why shouldn't you ask your professor what he or she means by thresholding? Commented Oct 14, 2013 at 21:25
• Thanks a lot. The label of the first line indicates the name of gene, but I do not know what the numbers mean either. Commented Oct 16, 2013 at 14:03
• This is from a online course material and I did not take this course. Anyway, I may try to email that professor to seek more information. Thanks Commented Oct 16, 2013 at 14:05

• There are two options: 'thresholding' may mean remove those rows/observations where any of the values is not in $[20,16000]$ or it could mean what you just described. I think the latter is used in statistics since it generates censored data, thus my answer. Commented Oct 14, 2013 at 21:12