# Optimal threshold from a uni-variate continuous variable without target

I have a univariate continuous variable. Now I would like to know is there any objective approach that can give me the optimal threshold that can help me make it a dichotomous variable with values as low and high. Please note that I have only variable (one column) and there isn't any dependent variable. The data looks like as shown below

Freq
31124
57643
70
13
1000


So now based on the data, I feel that terms occurring like 70 or 13 should be put under the "low" category because they can't really influence my output with such a little data whereas putting 1000 under low category may not be right because 1000 is a huge number and can influence the output.

Later, I might decide to exclude records from low category but that's a different discussion altogether.

Can help me how can we get an optimal threshold without having target variable?

• Dichotomization throws away a lot of information and is almost always a bad idea. (For instance, if you put your threshold below 1000, then you will treat 1000 and 31124 equally, since both are "high", although there is a much larger difference between the two than between 1000 and 70.) That said, to answer your question, you will need to think about how you can deduce whether one threshold is "better" than another one, so you will need to think about how you plan on using your thresholds. Jun 2, 2020 at 10:14
• Exactly. Based on your experience, can you let me know whether there is any theory/staistical approach which can let us know that any data that occurs only like 1 or 2% in your data, they can be removed. For ex: while dealing with missing data, we know that any variable which has around 70-80% of missing data can be removed without any second thought because they don't add any value to the output Jun 2, 2020 at 10:26
• You don't have sufficient basis for answering this question: the meaning of "optimal" implies you will use the data in some way in some analysis. If you could describe that intended analysis, it might be possible to supply objective answers.
– whuber
Jun 2, 2020 at 13:50
• @whuber - Basically I am trying to minimize the manual effort spent by reviewers ... I don't want them to review items which occurs very rarely and has very little importance ....I want the reviewers to focus only on terms which occur most, hence it's worth reviewing.... Jun 2, 2020 at 13:57
• IMHO, your current question has abstracted away the essence of that problem, making it impossible for the answers here to have any useful bearing on your original problem.
– whuber
Jun 2, 2020 at 13:59

In your case, it seems to me that Otsu's method should work well, which minimizes the within group variances, i.e. $$\sigma_{within}^2(t)=n_{low}(t)\cdot\sigma_{low}^2(t) + n_{high}(t)\cdot\sigma_{high}^2(t)$$ where $$t$$ is the threshold and "low" are the data values less than $$t$$ and "high" the data values greater than $$t$$. When you have $$n$$ data points, there are $$n-1$$ different possible values for $$t$$ and you can compute $$\sigma_{within}^2(t)$$ for each of them and pick the $$t$$ that minimizes it.