Skip to main content

Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms, but could also be used more generally in the sense of coarsening.

Filter by
Sorted by
Tagged with
0 votes
1 answer
20 views

How to model pixel shot noise

I'm interested in modelling the effect of shot noise on images. When taking a picture with a camera, the number of photons incident upon each pixel during the exposure time is (I believe) a ...
Yly's user avatar
  • 121
0 votes
0 answers
10 views

Modelling count from frequency

I have an empirically derived frequency curve. From the curve, I aim to model counts per unit time. The x-axis is diameter, and the y-axis the likelihood of observing the diameter per one year. I ...
Eron Raines's user avatar
1 vote
1 answer
58 views

I would like to perform a Kruskal-Wallis test in Jamovi [closed]

My problem is that I have data of sexual assertiveness (7 point Likert-scale, SAQ) and relationship satisfaction (5 point Likert-scale, RAS) but I would like to divide the sexual assertiveness ...
mse25's user avatar
  • 11
1 vote
0 answers
25 views

Test score equivalence - preparing data for Spearman correlation? [closed]

I have a somewhat theoretical question. I am trying to establish how closely scores across different language tests (IELTS, TOEFL, C1A, OET, DET) used in public domains match each other, given that ...
Amanda's user avatar
  • 11
0 votes
0 answers
38 views

Does replacing binned variables with Weight of Evidence values introduce data leakage?

In my company I've been noticing some binary classification modeling code that replaces bins of a continuous variable with the corresponding Weight of Evidence (WoE) of the given bin. As far as I ...
jglad's user avatar
  • 33
1 vote
1 answer
60 views

What distribution fit test to use for binned data?

I'm trying to compare two populations with 40 samples each. For each sample, I have two measurements of angle, measured in bins of 30 degrees (1-12), and I calculated the difference between the two (e....
Anthony BH's user avatar
2 votes
1 answer
167 views

Methods to derive cut-offs for continuous variables

I am working on a project to determine the variables that better predict the binary outcome. I am using conditional random forest and permimp::permimp for ...
Kate's user avatar
  • 185
0 votes
0 answers
33 views

Data driven approach to binning conditions based on a histogram

(Please note that this is all hypothetical at this point and the data specifics should not matter that much). Let's say I have a dataset where participants took a certain amount of time to complete a ...
grace.cutler's user avatar
1 vote
0 answers
33 views

Discretization in regression, experimentation, and causal inference as deafult [duplicate]

It crossed my mind that when designing an experiment and you're not interested in NHST but full regression model where coefficients for treatment exposure and relevant covariates are desired, perhaps ...
jbuddy_13's user avatar
  • 3,372
1 vote
1 answer
70 views

Does taking the ratio of Empirical Distributions (histogram bins) show their differences?

Background I have two Empirical distributions, both derived from social media data. The first represents a broad sample of ~4.8 million posts and the number of followers each post author has. The ...
Connor's user avatar
  • 635
0 votes
0 answers
40 views

The discretization of a continuous markov chain, is still a markov chain?

So I'm reading the Guihenneuc-Jouyaux & Robert paper from 1998 about discretization of Markov Chains in which they claim that a naive discretization of a continuous Markov chain is not usually a ...
nico's user avatar
  • 1
0 votes
0 answers
74 views

Cross-validation and automated binning of a continuous variable for a continuous target

I am building a pipeline in a machine learning project in which I would like to automatically discretize variables containing NAs. These NAs are justified in the context of the research and it is ...
hexolitemax's user avatar
1 vote
0 answers
104 views

How Minitab calculate optimize number of bins of a normal distribution histogram

I used minitab to draw a normal distribution histogram and saw that the number of bins in that graph doesn't follow any rules to calculate bin size that I know of (sturge, square-root, rice, freedman-...
nein_kariki's user avatar
0 votes
0 answers
31 views

The optimal way to stratify a numerical target variable into a categorical one for a machine learning algorithm

I have tabular data, the predictive variables are numerical and categorical and the target variable is a numerical one. Using the proper techniques I can make predictive models with R^2=0.95. Now let'...
Floralys's user avatar
0 votes
0 answers
39 views

Using maximally selected log rank to find two cutoffs instead of one

I have used the maximally selected log rank test to dichotomize a continuous variable (high vs low). I was curious if the maximally selected log rank test could be generalized to make 3 groups instead ...
Mike's user avatar
  • 111
0 votes
1 answer
147 views

WOE (Weight of evidence) cross-validation bias

I have a task to create credit scoring model using WOE encoding. I have a very small dataset, so I wont be able to perform testing on test and out-of-time samples. Thus, I am going to use cross-...
FedorT54's user avatar
0 votes
0 answers
36 views

Creating well-separated bins

I want to distribute my hospital data into 4 tiers (bins) based on the score. Tier 1 (good hospitals) has the highest score and Tier 4 (not so good hospitals) has the lowest score. The tiers can be of ...
museshad's user avatar
  • 101
0 votes
0 answers
21 views

Un-binning/Upsampling Ordinal Year-Bins into Individual Years for Random Forest Likert Analysis?

Question: Is it "quantitatively sound" to decompose/upsample year-bins (e.g., 2002-2006) into the component years when analyzing Likert Score data that was collected as a recollection of ...
stevelom's user avatar
6 votes
1 answer
602 views

Regression as classification: advantages?

I have read on many occasions deep learning practitioners recommending to treat regression problems (with continuous variables) as classification problems, by quantizing the output into bins and using ...
roygbiv's user avatar
  • 109
0 votes
1 answer
65 views

Descriptive statistics: a metric of the "lumpiness" of numeric vector

I have three datasets of continuous data. Is there a convenient metric for the "binnedness" of the data? How "lumpy" it is? I'd like a single number to allow me to distinguish ...
Carbonyl's user avatar
2 votes
2 answers
248 views

Quantize a continuous random variable

Suppose we have a continuous random variable $X$. We do not know its distribution function, but have $n$ i.i.d. samples. I am looking for methods that quantize (discretize) $X$ into a categorical ...
Mingzhou Liu's user avatar
0 votes
0 answers
130 views

Target binning in regression

I would like to find a predictive density for target variable via multi-class classification. Suppose we are given a set of features $\mathbf X$ and continuous target $\mathbf y$. Replace each $y$ ...
vladkkkkk's user avatar
  • 701
0 votes
0 answers
38 views

Binning continuous predictors, What is the best way? [duplicate]

I want to run a binary logistic regression to understanding (modeling) factors affecting nest-site selection in a bird species.. I think it is better if I transform continuous variables to categorical ...
Mostafa Ahmadi's user avatar
2 votes
1 answer
192 views

Regression With Binned Variables

Today our professor was discussing Sutradhar, Gu and Paszat (2016). In this paper the authors decided to study the relationships between patient characteristics and following the "advice" ...
stats_noob's user avatar
0 votes
0 answers
29 views

Using a variable as continuous and categorical

I have an IV and a DV which are both scale scores. I have categorized the IV and dependent variables based on previous studies. I wanted to do this to calculate the prevalence of DV and IV. I also ...
 Newton Chandra's user avatar
8 votes
3 answers
3k views

Why is it Bad to Discretize a Continuous Variable? [duplicate]

I am an MBA Student taking courses in statistics. I was doing some reading online and came across Carstensen (2020), which I unfortunately do not have access to. This article suggests that "...
stats_noob's user avatar
1 vote
1 answer
146 views

If I convert continuous data into ordinal discrete data, but number of the class is 100(1 class per 1 percentile), can I say this is 'continuous'?

I have some continuous data, and want to do kmeans clustering with this. But weirdly when I did kmean clustering with this data, the outcome was very conflicted with my presumption. So I decided to do ...
hogu's user avatar
  • 23
0 votes
0 answers
50 views

Binning: Selection of Range (Cutoff) for a Variable

In a current modeling work, we only use facilities with the secured portion % in the range of 70-100% (the range can be from 0-200%). These facilities (in the 70-100% secured percent range) are used ...
Vaibhav Kabdwal's user avatar
2 votes
1 answer
532 views

Methods to identify the optimal number of bins between two groups

I am training a supervised machine learning model. The training data contains 2 independent groups of people. The dataset contains independent continuous variables and 1 dependent binary variable. I ...
bgun's user avatar
  • 23
1 vote
0 answers
760 views

What's the best way to discretize a normal distribution?

I am working on a two-dimensional grid representing some physical properties at given spatial intervals. Let's say that I know there is a Gaussian distribution for $P(X=x \mid A=a)$, the probability ...
rocksNwaves's user avatar
1 vote
1 answer
18 views

Discretization Score test vs Training set

I'm in the process of training a NB model based on continuous features that need Equal Frequency Discretization to be used. Now, the question mark I'm facing is if discretization needs to be performed ...
davide cortellino's user avatar
1 vote
0 answers
51 views

Converting a Boolean relationship to the probability curve

I have two vectors, X and Y. Y[i] is related to X[i]. X is between 0 and 1 (it is a decimal number). Y is boolean, it is either 0 or 1. I want to calculate the probability of Y=1 for different X ...
poorya mirzavand's user avatar
0 votes
1 answer
42 views

How can I find the start of a range in an automated way?

I am looking at real estate sales data for different regions. I bin the data by sold price range in 1000 increments (see below). I am looking to find the start of the price range or cluster in which ...
Alex M's user avatar
  • 1
0 votes
1 answer
62 views

Converting a continuous output to risk score category and selecting the optimal number of bins

I am trying to convert a continuous measurement of a patient’s bone mineral density to a risk score which I will display to the user with the corresponding observed prevalence (observed probability of ...
Cicce19's user avatar
  • 35
1 vote
1 answer
792 views

How to setup the Panjer's recursion correctly?

I have a table of $k=(0,1,2,3,4,5,6)$ and $number=(40544,8082,1205,145,20,3,1)$ I need to fit data by a Compound Poisson-Gamma distribution and then make a discretization and compare results with ...
Nick's user avatar
  • 856
0 votes
0 answers
76 views

Is percentile a good method?

Hello I'm an undergraduate student doing research about prevalence of carpal tunnel syndrome among college students I want to follow the method of this research (prevalence of carpal tunnel syndrome ...
kareen kk's user avatar
1 vote
0 answers
107 views

Empirical distribution for feature binning

In paper "A simple yet effective baseline for non-attributed graph classification" (https://arxiv.org/pdf/1811.03508.pdf) authors use empirical distribution for feature binning. Precisely, ...
qalis's user avatar
  • 238
20 votes
2 answers
2k views

Why is Binning, Weight of Evidence and Information Value so ubiquitous in the Credit Risk/Finance industry?

In the credit risk industry (and finance industry as a whole, at least here in the UK), there is a very common and accepted 'proper' way to build scorecards. The general framework seems to be: ...
Liam Morgan's user avatar
0 votes
0 answers
297 views

Choose best binning for binned maximum likelihood fit?

I am trying to find the strength of signal over a background using a continuous variable, whose distributions are known for the expected signal, the expected background, and the observed data, along ...
dan's user avatar
  • 1
0 votes
2 answers
223 views

Combining quantile regression with binning

I'm trying to employ a framework where I uncover the marginal effects of the quantiles of one continuous variable on another continuous variable - something analogous to the Quantile-on-quantile (QQR) ...
Robert Brown's user avatar
1 vote
0 answers
24 views

Convert a distribution of a varialbe to another one

Assume an experiment where the observable is t and is related to another variable E with the following formula: $$E(t)=A/(t-t_0)^...
Thanos's user avatar
  • 121
0 votes
0 answers
54 views

Histogram where the bin heights take values into account

In a regular histogram the bin heights reflect the density of observations/data within them. Therefore, the areas of the bins on the chart reflect the quantity of observations they cover. They are for ...
user72845's user avatar
0 votes
0 answers
42 views

Which test to use in "Ordinal x Categorical" Data

I would like to ask a very fundamental question. First of all, I will simplify it to tell you the problem easily. Can I classify patients' ages into age groups (like <25, 25-35, 35-45, +45) and ...
Silefil's user avatar
1 vote
2 answers
299 views

Binning Calibrated probability scores for business use

Context: We have a model that outputs calibrated probability scores for a binary classification problem (events/nonevents). There is a general business requirement that we bin these outputs further to ...
PaulNoah's user avatar
2 votes
1 answer
92 views

Variance of 2D binned uniform density

I will start off by saying that I am aware of this post, but applying Sheppard's correction in my case would seem to lead to a negative variance; so either I don't understand that post, or I have a ...
Jonathan H's user avatar
0 votes
1 answer
420 views

Transform target/label variable into classes but classes are data-dependent: How to approach this correctly?

I was redirected from StackOverflow because my question is more about theory. I have a usual set-up with a pandas dataframe with some features and a numeric target variable (financial returns for ...
alphaH's user avatar
  • 37
1 vote
0 answers
271 views

What can we learn from binning before correlation?

I've seen a lot of questions about binning data and how it produces wrong results. "wrong" here refers to arbitrary increases in correlation strength, or sometimes even achieving two ...
mafu's user avatar
  • 143
1 vote
2 answers
4k views

How to calculate the mean from bin endpoints and frequencies? [duplicate]

Sometimes data extracted from reports do not have individual values, like 4, 23, 43, but grouped together like this: income level people in this group 10k to 20k 44 20k to 40k 240 40k to 80k 400 ...
samOz's user avatar
  • 11
2 votes
0 answers
48 views

When should I use a dummy variable?

I have data for neighbourhoods with median income. I also have a standard low-income cutoff. (Just using regression analysis) I feel it would lead to a simpler and cleaner result if I just use the ...
Dee's user avatar
  • 21
0 votes
0 answers
73 views

How do I bin my PCA results i.e., 120 equal bins of my large data set?

I am a new researcher in Japan and working on my project. I am lost after reading one paper, the paper is from very good general he uses the technique of binning of PCA but I don't know how what does ...
user_3264's user avatar
  • 101

1
2 3 4 5 6