Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms

55 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
116 views

How to compute optimal binning for two histograms

I am plotting two histograms on top of one another (using matplotlib, but that is tangential to my question). My current approach is to compute the mean of the optimal bin widths for each histogram ...
3
votes
0answers
2k views

Combining errors in a histogram (binned data)

I'm processing some data that requires binning before it goes through a regression algorithm. The script is in Python and uses the Numpy histogram function, but ...
2
votes
0answers
30 views

Whether and how to bin data beyond a certain threshold

I've seen many admonishments against binning and agreed with them in general, but I wonder it's advisable to do so under certain circumstances. To adapt an example from my actual project, let's say ...
2
votes
0answers
68 views

R Package for Optimal binning With Restrictions

Does anyone know about a R Package like SMBinning that can make Optimal Binning for WoE analysis where you can impose restrictions like minimum x number of Target in each bin monotome trend (WoE is ...
2
votes
0answers
202 views

What is the correct way to modify the bin-counts given a threshold for a chi-squared test?

When performing a chi-square test, one inputs the expected counts (via integrating probability distribution over respective bin bounds) and observed counts into the chi-square formula (denoted below). ...
2
votes
0answers
59 views

How do loan companies set interest rate tiers?

What statistical or machine learning methods do companies like Lending Club use to segment their customer base into loan grades A1-G5? What would a reasonable partitioning method look like after ...
2
votes
0answers
70 views

Given the number of Bins, what is the formula for estimating bin height in a 2D Histogram

Given the number of Bins, what is the formula for estimating bin height in a 2D Histogram in binormal distribution?
2
votes
0answers
71 views

References for the effect of binning methods?

There are two main ways to bin a numerical signal into discrete categorical values: Quintile Binning: Each Bin will have the same size Histogram Binning: A normal distribution will be used to bin the ...
2
votes
0answers
16 views

How to test if $I(\theta)\propto f(\theta)$?

How do I test if a finite resource is distributed across a population proportional to the density function for that population? Say $I(\theta)$ denotes resource allocation across $\theta$ (a known ...
2
votes
0answers
1k views

converting continous variable to discrete while maximizing gini

I have a continuous independent variable which is used to explain the dependent binary variable in logistic regression. The model user's requirement is to group this continuous variable to 6 bins. I ...
2
votes
0answers
530 views

A good alternative to data binning?

I read many times that data binning of continuous variables is a very bad idea. For instance, let's take something like heart rate and let's define the following 2 bins: (125 - 135), (136 - 145) ...
2
votes
0answers
425 views

On the uniform convergence of relative frequencies of events to their probabilities

I have read the article by Vapnik, Chervonenkis "On the uniform convergence of relative frequencies of events to their probabilities" Theory of Probability and Its Applications, vol XVI, n. , 1971. ...
1
vote
0answers
27 views

Combining categories by Weight of Evidence

When calculating Information Value and Weight of Evidence, it's possible to draw a chart of WoE for each variable to study its effect on the state of the target variable. Now, I know it's possible to ...
1
vote
0answers
93 views

Data binning with error bars

I've got some measured data where each data point has an uncertainty associated with it i.e, Data $\mathbf{x} = (x_1,x_2,...,x_n)$ with uncertainties $\mathbf{\alpha}_x = (\alpha_1,\alpha_2,...,\...
1
vote
0answers
170 views

How to fit a distribution to binned values that come from administrative data?

Fitting a distribution to data (e.g. with maximum likelihood), or testing goodness of fit (e.g. with Kolmogorov-Smirnov) assumes that the data are randomly drawn from a population. But what if the ...
1
vote
0answers
32 views

How creating bins for a numeric feature can enables the model to learn nonlinear relationships within a single feature?

I understood How binning of numerical feature would help build correlations between the feature & the predictor. For example For a regression problem, we can bucketize "population" feature into ...
1
vote
1answer
69 views

Should one binarize qualitative variables before applying a random forest?

On which of theses two kinds of sample would a Random Forest (and more precisely sklearn RandomForest algorithm) give the best results ? (Y and other_features are continuous numerical variables, and ...
1
vote
0answers
83 views

Discretize values/binning for missing data

I’m running an experiment where pairs make ratings after answering questions. There are 15 minute intervals where they attempt to answer 12 questions, but each pair makes it through a different number ...
1
vote
0answers
516 views

Why do these two logarithmic binning methods give different results? (in R)

Problem I am working on a problem involving logarithmic binning. I have a dataset consisting of individuals that have a size value and a production value. I am using an algorithm to logarithmically ...
1
vote
0answers
46 views

How should I formulate the problem of “binning” many millions of points into thousands of bins?

I'm working on a type of string matching problem. In this problem, I have ~10k strings in set A and ~25 million strings in set B. For each string in set B, I want to find the string in set A which is ...
1
vote
0answers
694 views

Binned residuals in logistic regression

I noticed that it's very hard to find something on binned residuals explained in a easy way except for this. This is my first time approaching to binned residuals. I understood that is not useful ...
1
vote
0answers
49 views

How are various rules for determining frequency distribution binwidth derived?

I've been looking up the problem of deciding appropriate binwidths for histograms and here's my broad-level understanding so far: If we have $n$ data points, we assume that they're realizations of $n$...
1
vote
0answers
76 views

Should I bin this continuous variable?

I understand that binning is generally frowned upon as you are "throwing away information". However, I'm not sure whether I should in this instance. I am running a logit regression. One of my ...
1
vote
0answers
109 views

Learning optimal histogram bins

I have a a data set containing of non-negative integer data for N subjects. In other words, each subject is represented by a vector of non negative integers (the vector length may vary from subject to ...
1
vote
0answers
563 views

Programmatically calculate which bin a value will fall into for a histogram

I'm trying to programmatically create a histogram. The number of bins is specified by the user. My issue in this problem is that I came up with what I thought was a reasonable way to determine which ...
1
vote
0answers
286 views

supervised binning

I have continuous outcome variable and continuous independent variable. I am trying to bin the independent variable that maximizes homogeneity within bins based on the outcome and maximize separation. ...
1
vote
0answers
190 views

How to compare binned CDFs?

We want to compare binned CDFs from sampled data. Each CDF is 101 numbers representing percentiles, i.e., $P(x<a_k)=k/100$ where $k=0..100$. The sample size is 1-3k (unknown at comparison time ...
1
vote
0answers
499 views

Binning Logistic Regression Results by Output in R

I have a logistic regression in R whose goal is to predict the probability of default on some test data. ...
1
vote
0answers
685 views

Supervised Binning with Naive Bayes

Context. I am working on a model to predict "churn". Subscription service where users pay a monthly fee to access the service. We would like to predict which accounts are likely to cancel or "churn". ...
1
vote
1answer
347 views

What are easy steps of finding cutpoint in continuous variable with Time to event outcome, in Stata?

I find it painful to manually guess a dichotomized cutpoint predictor (continuous) for an time to event outcome in Simple Cox proportional hazard model. Currently I was trying to find the cutpoint ...
1
vote
0answers
475 views

Comparing two sets of equal width binned data

I have two distinct sets of ~1500 intervals of differing lengths - let’s call them Interval Set 1 and Interval Set2. I have ...
1
vote
0answers
115 views

Find bins for observations where each bin has similar mean

I have 100000 observations with two variables on each, age on the range of 18-80 and interactions on the range of 1-1500. I want ...
1
vote
0answers
501 views

To BIN or not to BIN a continuous data to get a Fragment Size Distribution?

I have a data set in excel of almost 6000 entries (quantitative and continuous => P(X=x)=0, I mean the possible values for my continuous random variable X are uncountably many). Each point will ...
1
vote
0answers
70 views

How to select an error/weight for two-dimensional binned data?

Beginning with noisy data vectors $\mathbf{x}$ and $\mathbf{y}$, I have binned the data to vectors $\mathbf{x}_b$ and $\mathbf{y}_b$ of length $N_b$ with fixed linear ($\mathbf{x}_b^i - \mathbf{x}_b^{...
1
vote
0answers
613 views

selecting the bins for extremely skewed data

I have a data that exhibits nearly a power law distribution, and I want to know a good binning technique to summarize the statistics. For example consider the following data: $$ \begin{array}{rr} \...
1
vote
0answers
228 views

Is binning a skewed Likert scale variable justifiable?

I have recently advised some colleagues on the malpractice of binning a continuous variable, which was used in order to put it as a covariate in a regression model and retained as a significant ...
1
vote
0answers
649 views

Data Exploration - How To Bin Data?

There are 1100 data points split into about 35 discrete groups and distributed towards the left. After running a logistic regression which failed, then running it again by only predicting based on ...
1
vote
0answers
2k views

SAS optimal binning

In SAS enterprise miner we have the optimal binning feature which allows you to transform continuous variables into an ordered set of bins. The binning, as I read from one of their docs, is done so ...
0
votes
0answers
71 views
+50

Binning correlated variables after model fitting

Having read Frank Harrel's list of problem caused by binning continuous variables (http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous), I understand that binning should be avoided for model ...
0
votes
0answers
20 views

(How) Does different number of categories affect correlation?

I work with obesity in cats and many studies use a body condition score (BCS) to assess obesity. This is a somewhat subjective measure of how much fat covering an animal has. There are two commonly ...
0
votes
0answers
12 views

Comparing means in bins of different sample sizes

Suppose I have the following 2 plots showing the mean of a variable on a 2-d spatial grid, as well as the occurrence histogram (see bottom). Is there a specific way to compare these means, even ...
0
votes
0answers
23 views

problems in plotting decimal value distribution with bin width normalization in r using hist()

I am plotting data distribuiton. I get the expected plot with data which are only integer numbers. But I didn't get appropriate plot for decimal data sets. The code with integer numbers data as ...
0
votes
1answer
115 views

Optimal multivariate binning where the cut-points must be the same for all observations

I have a large data set with many discrete and continuous variables. All the variables are present in every observation. I want to explain (the log of) one continuous variable using all the other ...
0
votes
0answers
49 views

Statistical test for binned proportion data in JMP / MATLAB

I have been trying to determine the proper statistical test for comparing binned proportional data between groups. My data set is individual subjects (4-5 per group) with 76-165 cells per individual. ...
0
votes
0answers
33 views

How to correct make binning

I have a question regarding how to make binning correctly. I have 500 rows with different values (wages) which vary a lot. For example: in row number 20 the wage is \$600, whereas in 40 the wage is \...
0
votes
0answers
99 views

Distance Between 1-D Histograms and Binning

In order to calculate the distance between two 1-D histograms, they must have the same number of bins. I'm wondering if it's preferable to have more or fewer bins for such calculation. For example, ...
0
votes
0answers
784 views

Comparing the integral of a histogram and its distribution function

I asked a question here but I did a rather poor job of explaining my problem the question was poorly formed. If I have some histogram with $N_{total}$ number of total data points, and $N_{bins}$ ...
0
votes
0answers
990 views

Converting a continuous variable to categorical

I have several continuous predictor variables and one binary outcome variable. One of these predictor variables has the following description using the Hmisc package: ...
0
votes
0answers
28 views

Given a dataset and no information about its distribution, how to determine bin size?

Let's say I have a $k$-vector of numeric data $X \in \mathbb{R}^k$, and I want to plot a frequency distribution of all the data points in the vector. I've tried searching high and low for a method (...
0
votes
1answer
27 views

What influence do the sizes of the factor levels have in ANOVA?

I'd like to do an ANOVA on the following problem: The only dependent variable is the number of children a person has. The two independent variables are the person's age and the person's income. Of ...