1
$\begingroup$

I have a univariate continuous variable. Now I would like to know is there any objective approach that can give me the optimal threshold that can help me make it a dichotomous variable with values as low and high. Please note that I have only variable (one column) and there isn't any dependent variable. The data looks like as shown below

Freq
 31124
 57643 
 70
 13
 1000   

So now based on the data, I feel that terms occurring like 70 or 13 should be put under the "low" category because they can't really influence my output with such a little data whereas putting 1000 under low category may not be right because 1000 is a huge number and can influence the output.

Later, I might decide to exclude records from low category but that's a different discussion altogether.

Can help me how can we get an optimal threshold without having target variable?

$\endgroup$
6
  • $\begingroup$ Dichotomization throws away a lot of information and is almost always a bad idea. (For instance, if you put your threshold below 1000, then you will treat 1000 and 31124 equally, since both are "high", although there is a much larger difference between the two than between 1000 and 70.) That said, to answer your question, you will need to think about how you can deduce whether one threshold is "better" than another one, so you will need to think about how you plan on using your thresholds. $\endgroup$ Jun 2, 2020 at 10:14
  • $\begingroup$ Exactly. Based on your experience, can you let me know whether there is any theory/staistical approach which can let us know that any data that occurs only like 1 or 2% in your data, they can be removed. For ex: while dealing with missing data, we know that any variable which has around 70-80% of missing data can be removed without any second thought because they don't add any value to the output $\endgroup$
    – The Great
    Jun 2, 2020 at 10:26
  • $\begingroup$ You don't have sufficient basis for answering this question: the meaning of "optimal" implies you will use the data in some way in some analysis. If you could describe that intended analysis, it might be possible to supply objective answers. $\endgroup$
    – whuber
    Jun 2, 2020 at 13:50
  • $\begingroup$ @whuber - Basically I am trying to minimize the manual effort spent by reviewers ... I don't want them to review items which occurs very rarely and has very little importance ....I want the reviewers to focus only on terms which occur most, hence it's worth reviewing.... $\endgroup$
    – The Great
    Jun 2, 2020 at 13:57
  • $\begingroup$ IMHO, your current question has abstracted away the essence of that problem, making it impossible for the answers here to have any useful bearing on your original problem. $\endgroup$
    – whuber
    Jun 2, 2020 at 13:59

1 Answer 1

3
$\begingroup$

This is the classic "thresholding" problem as it occurs, e.g., in image processing. There the two classes are not "low" and "high", but "background" and "foreground". From a more general point of view, it is a clustering problem: how to divide the data into two groups by means of a threshold.

In your case, it seems to me that Otsu's method should work well, which minimizes the within group variances, i.e. $$\sigma_{within}^2(t)=n_{low}(t)\cdot\sigma_{low}^2(t) + n_{high}(t)\cdot\sigma_{high}^2(t)$$ where $t$ is the threshold and "low" are the data values less than $t$ and "high" the data values greater than $t$. When you have $n$ data points, there are $n-1$ different possible values for $t$ and you can compute $\sigma_{within}^2(t)$ for each of them and pick the $t$ that minimizes it.

$\endgroup$
3
  • $\begingroup$ In many applications this would be good advice, but it's not hard to identify situations where this approach is irrelevant or worse. $\endgroup$
    – whuber
    Jun 2, 2020 at 13:50
  • 1
    $\begingroup$ @whuber Yes, this is a good point: for any clustering algorithm with a fixed number of clusters k, it should be tested whether the choice for k is indeed appropriate. When,e.g., k = 2, the distribution should be tested for bimodality. I understood the question in such a way that there are good reasons, in this use case, to assume that there are two classes (low and high). $\endgroup$
    – cdalitz
    Jun 2, 2020 at 13:55
  • $\begingroup$ Hi Thanks for the response. Upvoted. $\endgroup$
    – The Great
    Jun 2, 2020 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.