The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

WoE optimal binning algorithm [closed]

I'm trying to translate this thing into a working algorithm. The paper here describes the algorithm implemented in MatLab. There is an alleged python translation here. but I never managed, or saw ...
1
vote
0answers
16 views

How to treat low frequency continuous variable in machine leanring

Hello I am working on machine learning model for count data, and I have various features that are highly skewed. The frequency table for one of the feature is given below. ...
0
votes
1answer
144 views

Optimal multivariate binning where the cut-points must be the same for all observations

I have a large data set with many discrete and continuous variables. All the variables are present in every observation. I want to explain (the log of) one continuous variable using all the other ...
3
votes
2answers
139 views

A data-independant transformation to discretize a range of values non-uniformly

I am sure this is trivial, but I am looking for a transformation that nonuniformly discretizes all values of a range into several bins. The bins should be variant and I'd like them to be smaller ...
0
votes
0answers
17 views

How to calculate errors for cumulative distribution function

I have some data points of the form $(x_i,y_i,\delta y_i)$, where $y$ are counts and the error associated to each $y_i = N$ is $y_i = \sqrt{N}$. I want to create the cumulative distribution of these ...
0
votes
1answer
24 views

Logistic regression interpretation in SPSS statistics

I observed a very strange behavior while doing logistic regression, univariance analysis and correlation analysis. I have dependent binary variable and several independent variables that should be ...
1
vote
0answers
41 views

How to properly bin the data for a fit

I am working on a spectroscopy project in which we adjust the wavelength of a laser and get some counts on the detector from some laser-atom interactions. The data that we have is in the form: $(\...
0
votes
0answers
101 views

Binning correlated variables after model fitting

Having read Frank Harrel's list of problem caused by binning continuous variables (http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous), I understand that binning should be avoided for model ...
3
votes
1answer
231 views

Multi Categorical Features vs multiple Features for categories

Say I am discretizing continuous data based on percentiles. (I realize this is generally frowned upon, but I am doing this for the sake of experiment) I am trying different percentiles, eg breaking ...
0
votes
1answer
28 views

What influence do the sizes of the factor levels have in ANOVA?

I'd like to do an ANOVA on the following problem: The only dependent variable is the number of children a person has. The two independent variables are the person's age and the person's income. Of ...
1
vote
1answer
353 views

What are easy steps of finding cutpoint in continuous variable with Time to event outcome, in Stata?

I find it painful to manually guess a dichotomized cutpoint predictor (continuous) for an time to event outcome in Simple Cox proportional hazard model. Currently I was trying to find the cutpoint ...
0
votes
0answers
612 views

Comparing bins (quintiles, deciles) based of $R^2$s from multivariate regression

I have a multivariate linear model: $\mathbf{Y} = \mathbf{X}\mathbf{B} + \mathbf{U}$ where the matrix $\mathbf{Y}$ represents stock returns, the design matrix is constituted by some explanatory ...
2
votes
2answers
81 views

good number of bins for logarithmic bin width

I was wondering how to estimate a good number of bins for my histogram. I know quite certainly that my data is well approximated by a LogNormal distribution. Previous studies have used logarithmic ...
1
vote
2answers
50 views

Is binning of continuous data always bad for statistical tests? [duplicate]

I was always thinking that binning of data if data is naturally continuous is bad. However, here is the case. The goal of study was to find if there is an association between a biomarker and disease ...
1
vote
1answer
80 views

Regression to Classification and back to Regression

Is it reasonable to transform regression problem into classification by binning target variable into classes and construct regression curve separately on each class?\ Precisely, if my goal is to ...
2
votes
1answer
20 views

How do you test for bias in a circular reference plane?

I've been trying to get my head around how to do hypothesis testing for a circular scale in hypothesis testing, but I am having a lot of trouble. I know well how to test for linear scales, but when it ...
0
votes
1answer
30 views

How to make SalePrice as a discrete value?

The target variable, Saleprice originally is a continuous value. I calculated ...
0
votes
0answers
20 views

(How) Does different number of categories affect correlation?

I work with obesity in cats and many studies use a body condition score (BCS) to assess obesity. This is a somewhat subjective measure of how much fat covering an animal has. There are two commonly ...
1
vote
0answers
31 views

Combining categories by Weight of Evidence

When calculating Information Value and Weight of Evidence, it's possible to draw a chart of WoE for each variable to study its effect on the state of the target variable. Now, I know it's possible to ...
0
votes
3answers
280 views

statsmodels logistic regression with binned variables has large coefficients and standard error for some variables

I'm fitting a logistic regression (binary) using Python's statsmodels, and here's a snippet of summary from the model: I have noticed that the large coefficients ...
0
votes
0answers
12 views

Comparing means in bins of different sample sizes

Suppose I have the following 2 plots showing the mean of a variable on a 2-d spatial grid, as well as the occurrence histogram (see bottom). Is there a specific way to compare these means, even ...
7
votes
1answer
951 views

How Do You Choose The Number of Bins To Use For A Chi-Squared GOF Test?

I'm working on developing a physics lab about radioactive decay, and in analyzing sample data I've taken, I ran into a statistics issue that surprised me. It is well known that the number of decays ...
0
votes
0answers
34 views

problems in plotting decimal value distribution with bin width normalization in r using hist()

I am plotting data distribuiton. I get the expected plot with data which are only integer numbers. But I didn't get appropriate plot for decimal data sets. The code with integer numbers data as ...
0
votes
0answers
121 views

binarization of variable - experimental threshold choice. Is it good approach?

I have some ratings averages values from 1 to 5(users were rating on 1,2,3,4,5 scale). I would like to split them into two classes: credible, ...
1
vote
1answer
69 views

Should one binarize qualitative variables before applying a random forest?

On which of theses two kinds of sample would a Random Forest (and more precisely sklearn RandomForest algorithm) give the best results ? (Y and other_features are continuous numerical variables, and ...
1
vote
0answers
30 views

What is the name for bins derived from Quantiles? [duplicate]

It's so easy to talk about "deciles" as if they were groups of observations that fall between the actual deciles, i.e., "any of the nine values that divide the sorted data into ten equal parts." But ...
1
vote
1answer
369 views

How to find statistical significance between binned data?

I have created a histogram of velocities for thousands of moving objects. I have bin sizes of 1 based on object weight. So bins 1-20, for weight 1gram to 20 grams. So that's the "x-axis". The y-axis, ...
2
votes
1answer
25 views

Equivalent of Kaplan Meier for an unbounded number of sets

I have used Kaplan-Meier method several times before when I compared how group $A$ survived compared to group $B$ through a period of (say) 5 years. Now I face a somewhat different scenario: I have ...
10
votes
5answers
3k views

Why should binning be avoided at all costs?

So I've read a few posts about why binning should always be avoided. A popular reference for that claim being this link. The main getaway being that the binning points (or cutpoints) are rather ...
8
votes
2answers
29k views

How to find average and median age from an aggregated frequency table

I am using excel and I am trying to find both the average age and median age. I have two columns. 1 for the category and the other for the number of people in each category. ...
6
votes
3answers
4k views

Logistic regression: categorical predictor vs. quantitative predictor

Why is it the case that when I run logistic regression with one categorical predictor, my regression is not significant whereas if I run the logistic regression with the same variable except it is ...
8
votes
2answers
3k views

What is the justification for unsupervised discretization of continuous variables?

A number of sources suggest that there are many negative consequences of the discretization (categorization) of continuous variables prior to statistical analysis (sample of references [1]-[4] below). ...
0
votes
1answer
510 views

Describing binned data?

Suppose you have some non-continuous data that you can bin, e.g. integer value test scores. So you go ahead and bin your data into bins of 100-90, 89-80, 79-70,...,9-0 and then you make a nice line ...
1
vote
2answers
564 views

Multinomial logistic regression, weighted logistic regression?

I have a binary predictor with many response variables. The binary predictor was originally continuous but was converted to binary ... if the response was $>1000$ then 1, else 0. I would like to ...
0
votes
0answers
50 views

Statistical test for binned proportion data in JMP / MATLAB

I have been trying to determine the proper statistical test for comparing binned proportional data between groups. My data set is individual subjects (4-5 per group) with 76-165 cells per individual. ...
1
vote
0answers
101 views

Data binning with error bars

I've got some measured data where each data point has an uncertainty associated with it i.e, Data $\mathbf{x} = (x_1,x_2,...,x_n)$ with uncertainties $\mathbf{\alpha}_x = (\alpha_1,\alpha_2,...,\...
1
vote
3answers
1k views

Creating a Predictive Model with Binned Data

I have a health dataset with the number of drinks per month someone consumes, and many other variables that are binned. For example, 1: income less than \$10000, 2=income less than \$20000, and so on. ...
0
votes
1answer
524 views

Logarithmic binning and log-normal distribution

I've an Italian cities dataset. It's similar to those British ones used in literature, but has some differences, though. I decided to perform a logarithmic binning to avoid noise on the right end of ...
1
vote
0answers
180 views

How to fit a distribution to binned values that come from administrative data?

Fitting a distribution to data (e.g. with maximum likelihood), or testing goodness of fit (e.g. with Kolmogorov-Smirnov) assumes that the data are randomly drawn from a population. But what if the ...
1
vote
0answers
33 views

How creating bins for a numeric feature can enables the model to learn nonlinear relationships within a single feature?

I understood How binning of numerical feature would help build correlations between the feature & the predictor. For example For a regression problem, we can bucketize "population" feature into ...
0
votes
0answers
1k views

Converting a continuous variable to categorical

I have several continuous predictor variables and one binary outcome variable. One of these predictor variables has the following description using the Hmisc package: ...
2
votes
1answer
97 views

Splitting range of independent variable to maximize prediction within the subranges

I have a dataset with two independent variables $X,Z \in \mathbb{R}$ and a dependent variable $Y \in \mathbb{R}$. This dataset has the following characteristics: given some number $z$ and a "small" $...
1
vote
2answers
790 views

Is there a general/golden rule for appropriate binning in a histogram?

I was wondering, is there a general rule or a "golden rule" that sets the appropriate bin size as a function of statistical parameters such as sample size, mean, median, mode, standard deviation, etc. ...
1
vote
1answer
355 views

Scott's and Freedman–Diaconis rules of the thumb for selecting bin width - disatvantages

Scott's and Freedman–Diaconis rules of the thumb are based on the following formula: In the article here it is stated that: While these appear to be useful estimates for unimodal densities ...
1
vote
0answers
86 views

Discretize values/binning for missing data

I’m running an experiment where pairs make ratings after answering questions. There are 15 minute intervals where they attempt to answer 12 questions, but each pair makes it through a different number ...
2
votes
0answers
30 views

Whether and how to bin data beyond a certain threshold

I've seen many admonishments against binning and agreed with them in general, but I wonder it's advisable to do so under certain circumstances. To adapt an example from my actual project, let's say ...
1
vote
1answer
30 views

Dichtomising continious treatment variables

In the past, I have repeatedly seen studies that dichtomize continuous treatment variables into a binary treatment dummy. I feel that this cannot be good practice. we loose important information we ...
2
votes
0answers
72 views

R Package for Optimal binning With Restrictions

Does anyone know about a R Package like SMBinning that can make Optimal Binning for WoE analysis where you can impose restrictions like minimum x number of Target in each bin monotome trend (WoE is ...
2
votes
2answers
15k views

Calculating the variance of the histogram of a grayscale image

I am doing image processing and I want to calculate the variance of a histogram of pixel intensities. The first method I have tried: The images store the pixels values using double precision numbers,...
111
votes
4answers
30k views

Assessing approximate distribution of data based on a histogram

Suppose I want to see whether my data is exponential based on a histogram (i.e. skewed to the right). Depending on how I group or bin the data, I can get wildly different histograms. One set of ...