Questions tagged [binning]
Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms, but could also be used more generally in the sense of coarsening.
249
questions
0
votes
0
answers
11
views
Using maximally selected log rank to find two cutoffs instead of one
I have used the maximally selected log rank test to dichotomize a continuous variable (high vs low). I was curious if the maximally selected log rank test could be generalized to make 3 groups instead ...
- 111
0
votes
1
answer
60
views
WOE (Weight of evidence) cross-validation bias
I have a task to create credit scoring model using WOE encoding. I have a very small dataset, so I wont be able to perform testing on test and out-of-time samples. Thus, I am going to use cross-...
- 33
0
votes
0
answers
36
views
Creating well-separated bins
I want to distribute my hospital data into 4 tiers (bins) based on the score. Tier 1 (good hospitals) has the highest score and Tier 4 (not so good hospitals) has the lowest score. The tiers can be of ...
- 101
0
votes
0
answers
13
views
Un-binning/Upsampling Ordinal Year-Bins into Individual Years for Random Forest Likert Analysis?
Question: Is it "quantitatively sound" to decompose/upsample year-bins (e.g., 2002-2006) into the component years when analyzing Likert Score data that was collected as a recollection of ...
- 1
5
votes
1
answer
159
views
Regression as classification: advantages?
I have read on many occasions deep learning practitioners recommending to treat regression problems (with continuous variables) as classification problems, by quantizing the output into bins and using ...
- 99
0
votes
1
answer
41
views
Descriptive statistics: a metric of the "lumpiness" of numeric vector
I have three datasets of continuous data. Is there a convenient metric for the "binnedness" of the data? How "lumpy" it is?
I'd like a single number to allow me to distinguish ...
- 1
1
vote
2
answers
106
views
Quantize a continuous random variable
Suppose we have a continuous random variable $X$. We do not know its distribution function, but have $n$ i.i.d. samples.
I am looking for methods that quantize (discretize) $X$ into a categorical ...
- 361
0
votes
0
answers
67
views
Target binning in regression
I would like to find a predictive density for target variable via multi-class classification. Suppose we are given a set of features $\mathbf X$ and continuous target $\mathbf y$. Replace each $y$ ...
- 661
0
votes
0
answers
14
views
Setting threshold with dynamic feedback
I have a dataset with 200,000 obs and the following variables:
score (continuous: 0-100), pred (binary: 0/1).
I want to create a binary variabel: pred2, that acts the following:
a) if score is high, ...
- 79
0
votes
0
answers
38
views
Binning continuous predictors, What is the best way? [duplicate]
I want to run a binary logistic regression to understanding (modeling) factors affecting nest-site selection in a bird species.. I think it is better if I transform continuous variables to categorical ...
- 139
2
votes
1
answer
126
views
Regression With Binned Variables
Today our professor was discussing Sutradhar, Gu and Paszat (2016). In this paper the authors decided to study the relationships between patient characteristics and following the "advice" ...
0
votes
0
answers
36
views
How can I calculate the entropy of a binned multivariate normal distribution?
I know that I can calculate the entropy of a multivariate distribution by using
$${N(\mu,{\boldsymbol {\Sigma }}) \\ \mathcal H(N) = \displaystyle {\frac {1}{2}}\ln \det \left(2\pi \mathrm {e} {\...
0
votes
0
answers
242
views
Optimal Binning for Continuous Variables in Clustering Analysis Python
I am creating a clustering model and would like to bin a set of continuous variables that hold profitability values for each account member for a given year-month. Are there any methods or python ...
- 11
8
votes
3
answers
2k
views
Why is it Bad to Discretize a Continuous Variable? [duplicate]
I am an MBA Student taking courses in statistics. I was doing some reading online and came across Carstensen (2020), which I unfortunately do not have access to.
This article suggests that "...
1
vote
1
answer
77
views
If I convert continuous data into ordinal discrete data, but number of the class is 100(1 class per 1 percentile), can I say this is 'continuous'?
I have some continuous data, and want to do kmeans clustering with this.
But weirdly when I did kmean clustering with this data, the outcome was very conflicted with my presumption. So I decided to do ...
- 23
0
votes
0
answers
41
views
Binning: Selection of Range (Cutoff) for a Variable
In a current modeling work, we only use facilities with the secured portion % in the range of 70-100% (the range can be from 0-200%). These facilities (in the 70-100% secured percent range) are used ...
2
votes
1
answer
296
views
Methods to identify the optimal number of bins between two groups
I am training a supervised machine learning model.
The training data contains 2 independent groups of people. The dataset contains independent continuous variables and 1 dependent binary variable.
I ...
- 23
1
vote
0
answers
346
views
What's the best way to discretize a normal distribution?
I am working on a two-dimensional grid representing some physical properties at given spatial intervals.
Let's say that I know there is a Gaussian distribution for $P(X=x \mid A=a)$, the probability ...
- 380
1
vote
1
answer
17
views
Discretization Score test vs Training set
I'm in the process of training a NB model based on continuous features that need Equal Frequency Discretization to be used.
Now, the question mark I'm facing is if discretization needs to be performed
...
1
vote
0
answers
50
views
Converting a Boolean relationship to the probability curve
I have two vectors, X and Y. Y[i] is related to X[i]. X is between 0 and 1 (it is a decimal number). Y is boolean, it is either 0 or 1. I want to calculate the probability of Y=1 for different X ...
0
votes
1
answer
39
views
How can I find the start of a range in an automated way?
I am looking at real estate sales data for different regions. I bin the data by sold price range in 1000 increments (see below). I am looking to find the start of the price range or cluster in which ...
- 1
0
votes
1
answer
54
views
Converting a continuous output to risk score category and selecting the optimal number of bins
I am trying to convert a continuous measurement of a patient’s bone mineral density to a risk score which I will display to the user with the corresponding observed prevalence (observed probability of ...
- 35
1
vote
1
answer
536
views
How to setup the Panjer's recursion correctly?
I have a table of $k=(0,1,2,3,4,5,6)$ and $number=(40544,8082,1205,145,20,3,1)$
I need to fit data by a Compound Poisson-Gamma distribution and then make a discretization and compare results with ...
- 848
0
votes
0
answers
72
views
Is percentile a good method?
Hello I'm an undergraduate student doing research about prevalence of carpal tunnel syndrome among college students I want to follow the method of this research (prevalence of carpal tunnel syndrome ...
- 71
1
vote
0
answers
91
views
Empirical distribution for feature binning
In paper "A simple yet effective baseline for non-attributed graph classification" (https://arxiv.org/pdf/1811.03508.pdf) authors use empirical distribution for feature binning. Precisely, ...
- 229
18
votes
2
answers
1k
views
Why is Binning, Weight of Evidence and Information Value so ubiquitous in the Credit Risk/Finance industry?
In the credit risk industry (and finance industry as a whole, at least here in the UK), there is a very common and accepted 'proper' way to build scorecards.
The general framework seems to be:
...
- 379
0
votes
0
answers
208
views
Choose best binning for binned maximum likelihood fit?
I am trying to find the strength of signal over a background using a continuous variable, whose distributions are known for the expected signal, the expected background, and the observed data, along ...
- 1
0
votes
2
answers
184
views
Combining quantile regression with binning
I'm trying to employ a framework where I uncover the marginal effects of the quantiles of one continuous variable on another continuous variable - something analogous to the Quantile-on-quantile (QQR) ...
- 206
1
vote
0
answers
22
views
Convert a distribution of a varialbe to another one
Assume an experiment where the observable is t and is related to another variable E with the following formula: $$E(t)=A/(t-t_0)^...
- 121
0
votes
0
answers
51
views
Histogram where the bin heights take values into account
In a regular histogram the bin heights reflect the density of observations/data within them. Therefore, the areas of the bins on the chart reflect the quantity of observations they cover. They are for ...
0
votes
0
answers
400
views
Number of bins for discretization
How do I decide on the right number of bins to discretize my continuous data? Are there are tests/techniques to do the same? Could someone give me some idea into existing approaches?
- 215
1
vote
2
answers
228
views
Binning Calibrated probability scores for business use
Context:
We have a model that outputs calibrated probability scores for a binary classification problem (events/nonevents). There is a general business requirement that we bin these outputs further to ...
- 11
1
vote
1
answer
76
views
Variance of 2D binned uniform density
I will start off by saying that I am aware of this post, but applying Sheppard's correction in my case would seem to lead to a negative variance; so either I don't understand that post, or I have a ...
- 266
0
votes
1
answer
328
views
Transform target/label variable into classes but classes are data-dependent: How to approach this correctly?
I was redirected from StackOverflow because my question is more about theory.
I have a usual set-up with a pandas dataframe with some features and a numeric target variable (financial returns for ...
- 37
1
vote
0
answers
218
views
What can we learn from binning before correlation?
I've seen a lot of questions about binning data and how it produces wrong results. "wrong" here refers to arbitrary increases in correlation strength, or sometimes even achieving two ...
- 123
1
vote
2
answers
3k
views
How to calculate the mean from bin endpoints and frequencies? [duplicate]
Sometimes data extracted from reports do not have individual values, like 4, 23, 43, but grouped together like this:
income level
people in this group
10k to 20k
44
20k to 40k
240
40k to 80k
400
...
- 11
2
votes
0
answers
45
views
When should I use a dummy variable?
I have data for neighbourhoods with median income. I also have a standard low-income cutoff. (Just using regression analysis) I feel it would lead to a simpler and cleaner result if I just use the ...
- 21
0
votes
0
answers
64
views
How do I bin my PCA results i.e., 120 equal bins of my large data set?
I am a new researcher in Japan and working on my project. I am lost after reading one paper, the paper is from very good general he uses the technique of binning of PCA but I don't know how what does ...
- 101
1
vote
1
answer
35
views
Logistic regression using a predictor that's part of the outcome
Say I have 2 continuous variables measuring the same thing (e.g., at-home blood pressure monitor and in-office blood pressure cuff with in-office measurements being the gold standard). At a cut-off of ...
2
votes
3
answers
348
views
Preferred way to sum different time series together (in software)
Is there a canonical/best approach to computationally summing different time series together?
What I mean by that is the operation
$$
\sum_i{s_i(t)} = S(t)
$$
where $s_i(t)$ is the $i$-th time series, ...
0
votes
0
answers
137
views
binning numerical variables?
I am dealing with a dataset composed of both numerical (discrete) and nominal variables and I have to classify a binary response.
Since the dataset is imbalanced, I decided to oversample the minority ...
- 11
2
votes
1
answer
7k
views
Difference between equal frequency and quantile binning
Equal-frequency binning divides the data set into bins that all have the same number of samples. Quantile binning assigns the same number of observations to each bin. What is the difference between ...
- 21
3
votes
1
answer
288
views
One Hot Encoding of ranges of data vs. leaving data as is for Logistic Regression
Recently whilst doing an assignment using the PIMA Diabetes set I ran Logistic Regression using, amongst others:
the age predictor as is
segmented the age into ranges and applied OHE (with and ...
- 245
0
votes
0
answers
131
views
Converting continuous predictor to category e.g. Age [duplicate]
I notice that on many examples one is keen to convert Age to a categorical age range.
I am wondering if that is always necessary.
The famous golf play decision tree example has ranges for temperatures ...
- 245
1
vote
0
answers
590
views
Modeling covariates in multiple regression
My aim is to find the association between intake of chocolate (continuous predictor) and blood pressure (continuous outcome) in a multiple linear regression. I have to include many covariates in order ...
- 11
0
votes
2
answers
4k
views
How can I determine the optimal binning system for a continuous variable in Python?
I've got two columns of data - a continuous variable that I'd like to treat as a categorical variable (i.e. bin it up), and a metric I want to measure by bin. Let's say the first column is income and ...
0
votes
1
answer
102
views
Plotting average shows (log) linear trend but fitting line has 0 p value
I have some data and I am examining the relationship between two variables. When I form x-bins and take summary statistics of y in those bins, I see the plot below. The black line is the mean of each ...
0
votes
0
answers
29
views
Which statistical techniques can be used to decide which definition of sets produces the most coherent grouping of data
To explain. I am a historian, and an almost complete statistical novice.
I am interested in exploring the ways in which generational
alignments might be identified, not via use of generational labels, ...
0
votes
0
answers
266
views
Uniform distribution in a logarithmic/isolethargic binning
Assume a variable $x$ follows a uniform distribution i.e. $P(x)-=const$.
In my case this is constant background as shown in the following figure with the green curve
This is a distribution with a ...
- 121
2
votes
1
answer
68
views
In practice, how to discretize continuous regressor with minimal impact on coefficient (or minimal information loss)?
Suppose I have some continuous data that looks like this (this is a mini example, not my real data):
...
- 2,516