# Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms, but could also be used more generally in the sense of coarsening.

264 questions
Filter by
Sorted by
Tagged with
20 views

### How to model pixel shot noise

I'm interested in modelling the effect of shot noise on images. When taking a picture with a camera, the number of photons incident upon each pixel during the exposure time is (I believe) a ...
• 121
10 views

### Modelling count from frequency

I have an empirically derived frequency curve. From the curve, I aim to model counts per unit time. The x-axis is diameter, and the y-axis the likelihood of observing the diameter per one year. I ...
1 vote
58 views

### I would like to perform a Kruskal-Wallis test in Jamovi [closed]

My problem is that I have data of sexual assertiveness (7 point Likert-scale, SAQ) and relationship satisfaction (5 point Likert-scale, RAS) but I would like to divide the sexual assertiveness ...
• 11
1 vote
25 views

### Test score equivalence - preparing data for Spearman correlation? [closed]

I have a somewhat theoretical question. I am trying to establish how closely scores across different language tests (IELTS, TOEFL, C1A, OET, DET) used in public domains match each other, given that ...
• 11
38 views

### Does replacing binned variables with Weight of Evidence values introduce data leakage?

In my company I've been noticing some binary classification modeling code that replaces bins of a continuous variable with the corresponding Weight of Evidence (WoE) of the given bin. As far as I ...
• 33
1 vote
60 views

### What distribution fit test to use for binned data?

I'm trying to compare two populations with 40 samples each. For each sample, I have two measurements of angle, measured in bins of 30 degrees (1-12), and I calculated the difference between the two (e....
167 views

### Methods to derive cut-offs for continuous variables

I am working on a project to determine the variables that better predict the binary outcome. I am using conditional random forest and permimp::permimp for ...
• 185
33 views

### Data driven approach to binning conditions based on a histogram

(Please note that this is all hypothetical at this point and the data specifics should not matter that much). Let's say I have a dataset where participants took a certain amount of time to complete a ...
• 113
1 vote
33 views

### Discretization in regression, experimentation, and causal inference as deafult [duplicate]

It crossed my mind that when designing an experiment and you're not interested in NHST but full regression model where coefficients for treatment exposure and relevant covariates are desired, perhaps ...
• 3,372
1 vote
70 views

### Does taking the ratio of Empirical Distributions (histogram bins) show their differences?

Background I have two Empirical distributions, both derived from social media data. The first represents a broad sample of ~4.8 million posts and the number of followers each post author has. The ...
• 635
40 views

### The discretization of a continuous markov chain, is still a markov chain?

So I'm reading the Guihenneuc-Jouyaux & Robert paper from 1998 about discretization of Markov Chains in which they claim that a naive discretization of a continuous Markov chain is not usually a ...
• 1
74 views

### Cross-validation and automated binning of a continuous variable for a continuous target

I am building a pipeline in a machine learning project in which I would like to automatically discretize variables containing NAs. These NAs are justified in the context of the research and it is ...
1 vote
104 views

### How Minitab calculate optimize number of bins of a normal distribution histogram

I used minitab to draw a normal distribution histogram and saw that the number of bins in that graph doesn't follow any rules to calculate bin size that I know of (sturge, square-root, rice, freedman-...
31 views

### The optimal way to stratify a numerical target variable into a categorical one for a machine learning algorithm

I have tabular data, the predictive variables are numerical and categorical and the target variable is a numerical one. Using the proper techniques I can make predictive models with R^2=0.95. Now let'...
• 21
39 views

### Using maximally selected log rank to find two cutoffs instead of one

I have used the maximally selected log rank test to dichotomize a continuous variable (high vs low). I was curious if the maximally selected log rank test could be generalized to make 3 groups instead ...
• 111
147 views

### WOE (Weight of evidence) cross-validation bias

I have a task to create credit scoring model using WOE encoding. I have a very small dataset, so I wont be able to perform testing on test and out-of-time samples. Thus, I am going to use cross-...
• 33
36 views

### Creating well-separated bins

I want to distribute my hospital data into 4 tiers (bins) based on the score. Tier 1 (good hospitals) has the highest score and Tier 4 (not so good hospitals) has the lowest score. The tiers can be of ...
• 101
21 views

### Un-binning/Upsampling Ordinal Year-Bins into Individual Years for Random Forest Likert Analysis?

Question: Is it "quantitatively sound" to decompose/upsample year-bins (e.g., 2002-2006) into the component years when analyzing Likert Score data that was collected as a recollection of ...
602 views

I have read on many occasions deep learning practitioners recommending to treat regression problems (with continuous variables) as classification problems, by quantizing the output into bins and using ...
• 109
65 views

### Descriptive statistics: a metric of the "lumpiness" of numeric vector

I have three datasets of continuous data. Is there a convenient metric for the "binnedness" of the data? How "lumpy" it is? I'd like a single number to allow me to distinguish ...
248 views

### Quantize a continuous random variable

Suppose we have a continuous random variable $X$. We do not know its distribution function, but have $n$ i.i.d. samples. I am looking for methods that quantize (discretize) $X$ into a categorical ...
• 523
130 views

### Target binning in regression

I would like to find a predictive density for target variable via multi-class classification. Suppose we are given a set of features $\mathbf X$ and continuous target $\mathbf y$. Replace each $y$ ...
• 701
38 views

### Binning continuous predictors, What is the best way? [duplicate]

I want to run a binary logistic regression to understanding (modeling) factors affecting nest-site selection in a bird species.. I think it is better if I transform continuous variables to categorical ...
192 views

### Regression With Binned Variables

Today our professor was discussing Sutradhar, Gu and Paszat (2016). In this paper the authors decided to study the relationships between patient characteristics and following the "advice" ...
29 views

### Using a variable as continuous and categorical

I have an IV and a DV which are both scale scores. I have categorized the IV and dependent variables based on previous studies. I wanted to do this to calculate the prevalence of DV and IV. I also ...
3k views

### Why is it Bad to Discretize a Continuous Variable? [duplicate]

I am an MBA Student taking courses in statistics. I was doing some reading online and came across Carstensen (2020), which I unfortunately do not have access to. This article suggests that "...
1 vote
146 views

### If I convert continuous data into ordinal discrete data, but number of the class is 100(1 class per 1 percentile), can I say this is 'continuous'?

I have some continuous data, and want to do kmeans clustering with this. But weirdly when I did kmean clustering with this data, the outcome was very conflicted with my presumption. So I decided to do ...
• 23
50 views

### Binning: Selection of Range (Cutoff) for a Variable

In a current modeling work, we only use facilities with the secured portion % in the range of 70-100% (the range can be from 0-200%). These facilities (in the 70-100% secured percent range) are used ...
532 views

### Methods to identify the optimal number of bins between two groups

I am training a supervised machine learning model. The training data contains 2 independent groups of people. The dataset contains independent continuous variables and 1 dependent binary variable. I ...
• 23
1 vote
760 views

### What's the best way to discretize a normal distribution?

I am working on a two-dimensional grid representing some physical properties at given spatial intervals. Let's say that I know there is a Gaussian distribution for $P(X=x \mid A=a)$, the probability ...
• 400
1 vote
18 views

### Discretization Score test vs Training set

I'm in the process of training a NB model based on continuous features that need Equal Frequency Discretization to be used. Now, the question mark I'm facing is if discretization needs to be performed ...
1 vote
51 views

### Converting a Boolean relationship to the probability curve

I have two vectors, X and Y. Y[i] is related to X[i]. X is between 0 and 1 (it is a decimal number). Y is boolean, it is either 0 or 1. I want to calculate the probability of Y=1 for different X ...
42 views

### How can I find the start of a range in an automated way?

I am looking at real estate sales data for different regions. I bin the data by sold price range in 1000 increments (see below). I am looking to find the start of the price range or cluster in which ...
62 views

### Converting a continuous output to risk score category and selecting the optimal number of bins

I am trying to convert a continuous measurement of a patient’s bone mineral density to a risk score which I will display to the user with the corresponding observed prevalence (observed probability of ...
• 35
1 vote
792 views

### How to setup the Panjer's recursion correctly?

I have a table of $k=(0,1,2,3,4,5,6)$ and $number=(40544,8082,1205,145,20,3,1)$ I need to fit data by a Compound Poisson-Gamma distribution and then make a discretization and compare results with ...
• 856
76 views

### Is percentile a good method?

Hello I'm an undergraduate student doing research about prevalence of carpal tunnel syndrome among college students I want to follow the method of this research (prevalence of carpal tunnel syndrome ...
1 vote
107 views

### Empirical distribution for feature binning

In paper "A simple yet effective baseline for non-attributed graph classification" (https://arxiv.org/pdf/1811.03508.pdf) authors use empirical distribution for feature binning. Precisely, ...
• 238
2k views

### Why is Binning, Weight of Evidence and Information Value so ubiquitous in the Credit Risk/Finance industry?

In the credit risk industry (and finance industry as a whole, at least here in the UK), there is a very common and accepted 'proper' way to build scorecards. The general framework seems to be: ...
• 479
297 views

### Choose best binning for binned maximum likelihood fit?

I am trying to find the strength of signal over a background using a continuous variable, whose distributions are known for the expected signal, the expected background, and the observed data, along ...
• 1
223 views

### Combining quantile regression with binning

I'm trying to employ a framework where I uncover the marginal effects of the quantiles of one continuous variable on another continuous variable - something analogous to the Quantile-on-quantile (QQR) ...
• 206
1 vote
24 views

### Convert a distribution of a varialbe to another one

Assume an experiment where the observable is t and is related to another variable E with the following formula: E(t)=A/(t-t_0)^...
• 121
54 views

### Histogram where the bin heights take values into account

In a regular histogram the bin heights reflect the density of observations/data within them. Therefore, the areas of the bins on the chart reflect the quantity of observations they cover. They are for ...
42 views

### Which test to use in "Ordinal x Categorical" Data

I would like to ask a very fundamental question. First of all, I will simplify it to tell you the problem easily. Can I classify patients' ages into age groups (like <25, 25-35, 35-45, +45) and ...
1 vote
299 views

### Binning Calibrated probability scores for business use

Context: We have a model that outputs calibrated probability scores for a binary classification problem (events/nonevents). There is a general business requirement that we bin these outputs further to ...
• 11
92 views

### Variance of 2D binned uniform density

I will start off by saying that I am aware of this post, but applying Sheppard's correction in my case would seem to lead to a negative variance; so either I don't understand that post, or I have a ...
• 316
420 views

### Transform target/label variable into classes but classes are data-dependent: How to approach this correctly?

I was redirected from StackOverflow because my question is more about theory. I have a usual set-up with a pandas dataframe with some features and a numeric target variable (financial returns for ...
• 37
1 vote
271 views

### What can we learn from binning before correlation?

I've seen a lot of questions about binning data and how it produces wrong results. "wrong" here refers to arbitrary increases in correlation strength, or sometimes even achieving two ...
• 143
1 vote
4k views

### How to calculate the mean from bin endpoints and frequencies? [duplicate]

Sometimes data extracted from reports do not have individual values, like 4, 23, 43, but grouped together like this: income level people in this group 10k to 20k 44 20k to 40k 240 40k to 80k 400 ...
• 11
48 views

### When should I use a dummy variable?

I have data for neighbourhoods with median income. I also have a standard low-income cutoff. (Just using regression analysis) I feel it would lead to a simpler and cleaner result if I just use the ...
• 21