Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms

61 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
118 views

How to compute optimal binning for two histograms

I am plotting two histograms on top of one another (using matplotlib, but that is tangential to my question). My current approach is to compute the mean of the optimal bin widths for each histogram ...
3
votes
0answers
2k views

Combining errors in a histogram (binned data)

I'm processing some data that requires binning before it goes through a regression algorithm. The script is in Python and uses the Numpy histogram function, but ...
2
votes
0answers
13 views

Predicting binary outcomes for observations given statistics on binned data

SAT Verbal scores range from 200 to 800 in increments of 10. MIT says that for the class of 2023, the acceptance rates were, for various score ranges 750-800 10% = 677/6504 700-740 06% = 312/5039 ...
2
votes
0answers
32 views

Whether and how to bin data beyond a certain threshold

I've seen many admonishments against binning and agreed with them in general, but I wonder it's advisable to do so under certain circumstances. To adapt an example from my actual project, let's say ...
2
votes
0answers
78 views

R Package for Optimal binning With Restrictions

Does anyone know about a R Package like SMBinning that can make Optimal Binning for WoE analysis where you can impose restrictions like minimum x number of Target in each bin monotome trend (WoE is ...
2
votes
0answers
233 views

What is the correct way to modify the bin-counts given a threshold for a chi-squared test?

When performing a chi-square test, one inputs the expected counts (via integrating probability distribution over respective bin bounds) and observed counts into the chi-square formula (denoted below). ...
2
votes
0answers
60 views

How do loan companies set interest rate tiers?

What statistical or machine learning methods do companies like Lending Club use to segment their customer base into loan grades A1-G5? What would a reasonable partitioning method look like after ...
2
votes
0answers
87 views

Given the number of Bins, what is the formula for estimating bin height in a 2D Histogram

Given the number of Bins, what is the formula for estimating bin height in a 2D Histogram in binormal distribution?
2
votes
0answers
72 views

References for the effect of binning methods?

There are two main ways to bin a numerical signal into discrete categorical values: Quintile Binning: Each Bin will have the same size Histogram Binning: A normal distribution will be used to bin the ...
2
votes
0answers
16 views

How to test if $I(\theta)\propto f(\theta)$?

How do I test if a finite resource is distributed across a population proportional to the density function for that population? Say $I(\theta)$ denotes resource allocation across $\theta$ (a known ...
2
votes
0answers
1k views

converting continous variable to discrete while maximizing gini

I have a continuous independent variable which is used to explain the dependent binary variable in logistic regression. The model user's requirement is to group this continuous variable to 6 bins. I ...
2
votes
0answers
614 views

Knuth rule for number of bins of a histogram vs. chi2 fitting

I try to make a histogram and then fit some distribution to it by means of chi2. The Knuth rule (I have some bimodal cases so I'm not using Freedman-Diaconis or Scott) gives me the following histogram ...
2
votes
0answers
553 views

A good alternative to data binning?

I read many times that data binning of continuous variables is a very bad idea. For instance, let's take something like heart rate and let's define the following 2 bins: (125 - 135), (136 - 145) ...
2
votes
0answers
479 views

On the uniform convergence of relative frequencies of events to their probabilities

I have read the article by Vapnik, Chervonenkis "On the uniform convergence of relative frequencies of events to their probabilities" Theory of Probability and Its Applications, vol XVI, n. , 1971. ...
1
vote
0answers
24 views

How to treat low frequency continuous variable in machine leanring

Hello I am working on machine learning model for count data, and I have various features that are highly skewed. The frequency table for one of the feature is given below. ...
1
vote
0answers
45 views

How to properly bin the data for a fit

I am working on a spectroscopy project in which we adjust the wavelength of a laser and get some counts on the detector from some laser-atom interactions. The data that we have is in the form: $(\...
1
vote
0answers
72 views

Combining categories by Weight of Evidence

When calculating Information Value and Weight of Evidence, it's possible to draw a chart of WoE for each variable to study its effect on the state of the target variable. Now, I know it's possible to ...
1
vote
0answers
123 views

Data binning with error bars

I've got some measured data where each data point has an uncertainty associated with it i.e, Data $\mathbf{x} = (x_1,x_2,...,x_n)$ with uncertainties $\mathbf{\alpha}_x = (\alpha_1,\alpha_2,...,\...
1
vote
0answers
224 views

How to fit a distribution to binned values that come from administrative data?

Fitting a distribution to data (e.g. with maximum likelihood), or testing goodness of fit (e.g. with Kolmogorov-Smirnov) assumes that the data are randomly drawn from a population. But what if the ...
1
vote
0answers
38 views

How creating bins for a numeric feature can enables the model to learn nonlinear relationships within a single feature?

I understood How binning of numerical feature would help build correlations between the feature & the predictor. For example For a regression problem, we can bucketize "population" feature into ...
1
vote
1answer
71 views

Should one binarize qualitative variables before applying a random forest?

On which of theses two kinds of sample would a Random Forest (and more precisely sklearn RandomForest algorithm) give the best results ? (Y and other_features are continuous numerical variables, and ...
1
vote
0answers
92 views

Discretize values/binning for missing data

I’m running an experiment where pairs make ratings after answering questions. There are 15 minute intervals where they attempt to answer 12 questions, but each pair makes it through a different number ...
1
vote
0answers
668 views

Why do these two logarithmic binning methods give different results? (in R)

Problem I am working on a problem involving logarithmic binning. I have a dataset consisting of individuals that have a size value and a production value. I am using an algorithm to logarithmically ...
1
vote
0answers
50 views

How should I formulate the problem of “binning” many millions of points into thousands of bins?

I'm working on a type of string matching problem. In this problem, I have ~10k strings in set A and ~25 million strings in set B. For each string in set B, I want to find the string in set A which is ...
1
vote
0answers
812 views

Binned residuals in logistic regression

I noticed that it's very hard to find something on binned residuals explained in a easy way except for this. This is my first time approaching to binned residuals. I understood that is not useful ...
1
vote
0answers
53 views

How are various rules for determining frequency distribution binwidth derived?

I've been looking up the problem of deciding appropriate binwidths for histograms and here's my broad-level understanding so far: If we have $n$ data points, we assume that they're realizations of $n$...
1
vote
0answers
79 views

Should I bin this continuous variable?

I understand that binning is generally frowned upon as you are "throwing away information". However, I'm not sure whether I should in this instance. I am running a logit regression. One of my ...
1
vote
0answers
112 views

Learning optimal histogram bins

I have a a data set containing of non-negative integer data for N subjects. In other words, each subject is represented by a vector of non negative integers (the vector length may vary from subject to ...
1
vote
0answers
644 views

Programmatically calculate which bin a value will fall into for a histogram

I'm trying to programmatically create a histogram. The number of bins is specified by the user. My issue in this problem is that I came up with what I thought was a reasonable way to determine which ...
1
vote
0answers
312 views

supervised binning

I have continuous outcome variable and continuous independent variable. I am trying to bin the independent variable that maximizes homogeneity within bins based on the outcome and maximize separation. ...
1
vote
0answers
195 views

How to compare binned CDFs?

We want to compare binned CDFs from sampled data. Each CDF is 101 numbers representing percentiles, i.e., $P(x<a_k)=k/100$ where $k=0..100$. The sample size is 1-3k (unknown at comparison time ...
1
vote
0answers
518 views

Binning Logistic Regression Results by Output in R

I have a logistic regression in R whose goal is to predict the probability of default on some test data. ...
1
vote
0answers
718 views

Supervised Binning with Naive Bayes

Context. I am working on a model to predict "churn". Subscription service where users pay a monthly fee to access the service. We would like to predict which accounts are likely to cancel or "churn". ...
1
vote
1answer
371 views

What are easy steps of finding cutpoint in continuous variable with Time to event outcome, in Stata?

I find it painful to manually guess a dichotomized cutpoint predictor (continuous) for an time to event outcome in Simple Cox proportional hazard model. Currently I was trying to find the cutpoint ...
1
vote
0answers
510 views

Comparing two sets of equal width binned data

I have two distinct sets of ~1500 intervals of differing lengths - let’s call them Interval Set 1 and Interval Set2. I have ...
1
vote
0answers
121 views

Find bins for observations where each bin has similar mean

I have 100000 observations with two variables on each, age on the range of 18-80 and interactions on the range of 1-1500. I want ...
1
vote
0answers
502 views

To BIN or not to BIN a continuous data to get a Fragment Size Distribution?

I have a data set in excel of almost 6000 entries (quantitative and continuous => P(X=x)=0, I mean the possible values for my continuous random variable X are uncountably many). Each point will ...
1
vote
0answers
71 views

How to select an error/weight for two-dimensional binned data?

Beginning with noisy data vectors $\mathbf{x}$ and $\mathbf{y}$, I have binned the data to vectors $\mathbf{x}_b$ and $\mathbf{y}_b$ of length $N_b$ with fixed linear ($\mathbf{x}_b^i - \mathbf{x}_b^{...
1
vote
0answers
647 views

selecting the bins for extremely skewed data

I have a data that exhibits nearly a power law distribution, and I want to know a good binning technique to summarize the statistics. For example consider the following data: $$ \begin{array}{rr} \...
1
vote
0answers
241 views

Is binning a skewed Likert scale variable justifiable?

I have recently advised some colleagues on the malpractice of binning a continuous variable, which was used in order to put it as a covariate in a regression model and retained as a significant ...
1
vote
0answers
654 views

Data Exploration - How To Bin Data?

There are 1100 data points split into about 35 discrete groups and distributed towards the left. After running a logistic regression which failed, then running it again by only predicting based on ...
1
vote
0answers
2k views

SAS optimal binning

In SAS enterprise miner we have the optimal binning feature which allows you to transform continuous variables into an ordered set of bins. The binning, as I read from one of their docs, is done so ...
0
votes
0answers
16 views

Using Binning before Mann-Whitney for Temperature Data

I have daily temperature for 2 cities and I am trying to see if we can conclude that one city is warmer than the other. I could use a Mann-Whitney for a whole year or I can bin the temperature into ...
0
votes
0answers
19 views

How to calculate errors for cumulative distribution function

I have some data points of the form $(x_i,y_i,\delta y_i)$, where $y$ are counts and the error associated to each $y_i = N$ is $y_i = \sqrt{N}$. I want to create the cumulative distribution of these ...
0
votes
1answer
29 views

Logistic regression interpretation in SPSS statistics

I observed a very strange behavior while doing logistic regression, univariance analysis and correlation analysis. I have dependent binary variable and several independent variables that should be ...
0
votes
0answers
102 views

Binning correlated variables after model fitting

Having read Frank Harrel's list of problem caused by binning continuous variables (http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous), I understand that binning should be avoided for model ...
0
votes
0answers
22 views

(How) Does different number of categories affect correlation?

I work with obesity in cats and many studies use a body condition score (BCS) to assess obesity. This is a somewhat subjective measure of how much fat covering an animal has. There are two commonly ...
0
votes
0answers
12 views

Comparing means in bins of different sample sizes

Suppose I have the following 2 plots showing the mean of a variable on a 2-d spatial grid, as well as the occurrence histogram (see bottom). Is there a specific way to compare these means, even ...
0
votes
0answers
51 views

problems in plotting decimal value distribution with bin width normalization in r using hist()

I am plotting data distribuiton. I get the expected plot with data which are only integer numbers. But I didn't get appropriate plot for decimal data sets. The code with integer numbers data as ...
0
votes
0answers
59 views

Statistical test for binned proportion data in JMP / MATLAB

I have been trying to determine the proper statistical test for comparing binned proportional data between groups. My data set is individual subjects (4-5 per group) with 76-165 cells per individual. ...