Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms

0
votes
0answers
17 views

problems in plotting decimal value distribution with bin width normalization in r using hist()

I am plotting data distribuiton. I get the expected plot with data which are only integer numbers. But I didn't get appropriate plot for decimal data sets. The code with integer numbers data as ...
1
vote
0answers
29 views

What is the name for bins derived from Quantiles? [duplicate]

It's so easy to talk about "deciles" as if they were groups of observations that fall between the actual deciles, i.e., "any of the nine values that divide the sorted data into ten equal parts." But ...
2
votes
1answer
20 views

Equivalent of Kaplan Meier for an unbounded number of sets

I have used Kaplan-Meier method several times before when I compared how group $A$ survived compared to group $B$ through a period of (say) 5 years. Now I face a somewhat different scenario: I have ...
10
votes
5answers
3k views

Why should binning be avoided at all costs?

So I've read a few posts about why binning should always be avoided. A popular reference for that claim being this link. The main getaway being that the binning points (or cutpoints) are rather ...
0
votes
2answers
194 views

statsmodels logistic regression with binned variables has large coefficients and standard error for some variables

I'm fitting a logistic regression (binary) using Python's statsmodels, and here's a snippet of summary from the model: I have noticed that the large coefficients ...
0
votes
1answer
52 views

Optimal multivariate binning where the cut-points must be the same for all observations

I have a large data set with many discrete and continuous variables. All the variables are present in every observation. I want to explain (the log of) one continuous variable using all the other ...
0
votes
0answers
44 views

Statistical test for binned proportion data in JMP / MATLAB

I have been trying to determine the proper statistical test for comparing binned proportional data between groups. My data set is individual subjects (4-5 per group) with 76-165 cells per individual. ...
2
votes
0answers
43 views

Data binning with error bars

I've got some measured data where each data point has an uncertainty associated with it i.e, Data $\mathbf{x} = (x_1,x_2,...,x_n)$ with uncertainties $\mathbf{\alpha}_x = (\alpha_1,\alpha_2,...,\...
1
vote
0answers
113 views

How to fit a distribution to binned values that come from administrative data?

Fitting a distribution to data (e.g. with maximum likelihood), or testing goodness of fit (e.g. with Kolmogorov-Smirnov) assumes that the data are randomly drawn from a population. But what if the ...
1
vote
0answers
23 views

How creating bins for a numeric feature can enables the model to learn nonlinear relationships within a single feature?

I understood How binning of numerical feature would help build correlations between the feature & the predictor. For example For a regression problem, we can bucketize "population" feature into ...
2
votes
1answer
94 views

Splitting range of independent variable to maximize prediction within the subranges

I have a dataset with two independent variables $X,Z \in \mathbb{R}$ and a dependent variable $Y \in \mathbb{R}$. This dataset has the following characteristics: given some number $z$ and a "small" $...
1
vote
1answer
238 views

Scott's and Freedman–Diaconis rules of the thumb for selecting bin width - disatvantages

Scott's and Freedman–Diaconis rules of the thumb are based on the following formula: In the article here it is stated that: While these appear to be useful estimates for unimodal densities ...
1
vote
1answer
65 views

Should one binarize qualitative variables before applying a random forest?

On which of theses two kinds of sample would a Random Forest (and more precisely sklearn RandomForest algorithm) give the best results ? (Y and other_features are continuous numerical variables, and ...
1
vote
0answers
61 views

Discretize values/binning for missing data

I’m running an experiment where pairs make ratings after answering questions. There are 15 minute intervals where they attempt to answer 12 questions, but each pair makes it through a different number ...
2
votes
0answers
27 views

Whether and how to bin data beyond a certain threshold

I've seen many admonishments against binning and agreed with them in general, but I wonder it's advisable to do so under certain circumstances. To adapt an example from my actual project, let's say ...
0
votes
1answer
326 views

Logarithmic binning and log-normal distribution

I've an Italian cities dataset. It's similar to those British ones used in literature, but has some differences, though. I decided to perform a logarithmic binning to avoid noise on the right end of ...
1
vote
1answer
28 views

Dichtomising continious treatment variables

In the past, I have repeatedly seen studies that dichtomize continuous treatment variables into a binary treatment dummy. I feel that this cannot be good practice. we loose important information we ...
0
votes
0answers
47 views

Binning Continuous variable

I have a data on weekly exports. The exports are generally of the order of 0 or 0.3 million,0.6 million,0.9 million as can be observed from their histograms which have a high peaks at about these ...
2
votes
0answers
62 views

R Package for Optimal binning With Restrictions

Does anyone know about a R Package like SMBinning that can make Optimal Binning for WoE analysis where you can impose restrictions like minimum x number of Target in each bin monotome trend (WoE is ...
-1
votes
0answers
19 views

How to choose time bin width for Poisson Process?

Newbie here - just started learning more about poisson distribution. I have a time series of occurrence of events, and I thought this might fit a Poisson process. Before I do hypothesis testing, I ...
0
votes
0answers
97 views

For a histogram, is there any standard rule about how large the min and max bin should be (underflow bin and overflow bin)?

Working with several large datasets in excel, about N=900. I would like to divide the data into a number of categories corresponding to low, med, high etc. for this biological variable. Histograms ...
0
votes
0answers
26 views

How to correct make binning

I have a question regarding how to make binning correctly. I have 500 rows with different values (wages) which vary a lot. For example: in row number 20 the wage is \$600, whereas in 40 the wage is \...
0
votes
1answer
52 views

Why is it useful to have the same amount of data in each level of a categorical variable?

Something my lecturer said but I can't find why this is the case. I have to make a continuous variable into a categorical one, and the data is left skewed. Is it more important to have equal ranges or ...
2
votes
1answer
106 views

Learning a continuous model from binned data

A very similar question has been asked before, but it didn't get a real answer. Background I would like to develop a probability model for a continuous, ratio-scale random variable $Y$. Let's say it ...
0
votes
0answers
87 views

Distance Between 1-D Histograms and Binning

In order to calculate the distance between two 1-D histograms, they must have the same number of bins. I'm wondering if it's preferable to have more or fewer bins for such calculation. For example, ...
7
votes
3answers
412 views

Can the 'bin size' in a histogram be thought of as a regularity constraint?

When thinking about a histogram as an estimate of the density function, is it reasonable to think of the bin size as a parameter that constrains the local structure of that function? Also, is there a ...
2
votes
0answers
172 views

What is the correct way to modify the bin-counts given a threshold for a chi-squared test?

When performing a chi-square test, one inputs the expected counts (via integrating probability distribution over respective bin bounds) and observed counts into the chi-square formula (denoted below). ...
1
vote
0answers
362 views

Why do these two logarithmic binning methods give different results? (in R)

Problem I am working on a problem involving logarithmic binning. I have a dataset consisting of individuals that have a size value and a production value. I am using an algorithm to logarithmically ...
1
vote
0answers
42 views

How should I formulate the problem of “binning” many millions of points into thousands of bins?

I'm working on a type of string matching problem. In this problem, I have ~10k strings in set A and ~25 million strings in set B. For each string in set B, I want to find the string in set A which is ...
0
votes
0answers
598 views

Comparing the integral of a histogram and its distribution function

I asked a question here but I did a rather poor job of explaining my problem the question was poorly formed. If I have some histogram with $N_{total}$ number of total data points, and $N_{bins}$ ...
3
votes
0answers
104 views

How to compute optimal binning for two histograms

I am plotting two histograms on top of one another (using matplotlib, but that is tangential to my question). My current approach is to compute the mean of the optimal bin widths for each histogram ...
6
votes
1answer
110 views

Where can I find more materials on 'binning' after PCA?

I am supposed to write a literature review on a particular paper for my University and I am lost after reading the main paper I am supposed to read. The link to the paper is here. The paper is from ...
8
votes
1answer
715 views

How Do You Choose The Number of Bins To Use For A Chi-Squared GOF Test?

I'm working on developing a physics lab about radioactive decay, and in analyzing sample data I've taken, I ran into a statistics issue that surprised me. It is well known that the number of decays ...
0
votes
1answer
159 views

Defining bin sizes for plotting histograms [duplicate]

Perhaps a basic question, but is there a method or definition of how to choose the optimum bin size for some data with the intent to plot as a histogram? At present the best option I can think of is ...
1
vote
0answers
602 views

Binned residuals in logistic regression

I noticed that it's very hard to find something on binned residuals explained in a easy way except for this. This is my first time approaching to binned residuals. I understood that is not useful ...
0
votes
1answer
49 views

Is it incorrect to interpolate between known probabilities for binned data?

I have 7 bins of binary data according to a variable x as follows: x1: N=1000, success = 324, failure=676 x2: N=1000, success = 444, failure= 556 .... .... x7: N=...., success = ..., failure = ......
0
votes
1answer
200 views

Normal distribution applied to oil prices

I'm taking an introduction to statistics course and in attempt to reinforce what I have learned I have fit normal distributions to oil open,high,low,close prices over past 10 years where the number of ...
1
vote
1answer
129 views

R: Automatically group the insignificant dummy levels and re-fit the model

I am running a reg model with Weekdays as my dummy variables so I can find the weekday effects to the output metric. The picture below shows the results from this regression, and only Sunday is ...
0
votes
0answers
937 views

Converting a continuous variable to categorical

I have several continuous predictor variables and one binary outcome variable. One of these predictor variables has the following description using the Hmisc package: ...
0
votes
1answer
376 views

Discretizing/Binning Continuous Variable by Continuous Response

I am trying to discretize a continuous variable that does not follow a linear relationship with our response variable. I am trying to find the optimal way to discretize this variable to better ...
1
vote
3answers
947 views

Creating a Predictive Model with Binned Data

I have a health dataset with the number of drinks per month someone consumes, and many other variables that are binned. For example, 1: income less than \$10000, 2=income less than \$20000, and so on. ...
1
vote
0answers
43 views

How are various rules for determining frequency distribution binwidth derived?

I've been looking up the problem of deciding appropriate binwidths for histograms and here's my broad-level understanding so far: If we have $n$ data points, we assume that they're realizations of $n$...
0
votes
0answers
27 views

Given a dataset and no information about its distribution, how to determine bin size?

Let's say I have a $k$-vector of numeric data $X \in \mathbb{R}^k$, and I want to plot a frequency distribution of all the data points in the vector. I've tried searching high and low for a method (...
1
vote
2answers
1k views

RandomForest( ) more than 32factor levels [closed]

I used randomForest( ) function in R. But R can't deal variables that more 32factor levels. But my factor variables are very important thing. So I want to use this variable. Then, how to deal this ...
1
vote
0answers
72 views

Should I bin this continuous variable?

I understand that binning is generally frowned upon as you are "throwing away information". However, I'm not sure whether I should in this instance. I am running a logit regression. One of my ...
0
votes
1answer
460 views

Describing binned data?

Suppose you have some non-continuous data that you can bin, e.g. integer value test scores. So you go ahead and bin your data into bins of 100-90, 89-80, 79-70,...,9-0 and then you make a nice line ...
2
votes
1answer
2k views

Characteristic of good binning for weight of evidence algorithm

I am using logistic regression for classification purpose. For reduction of features and better precision I am using Weight of evidence technique. Also I need to use python for this. As there is no ...
1
vote
1answer
843 views

Bins in Regression Discontinuity Designs

Lee and Lemieux (p. 31, 2009) suggest the researcher to also present graphs while doing Regression discontinuity design analysis. They suggest the following procedure: "...for some bandwidth $h$, ...
2
votes
0answers
56 views

How do loan companies set interest rate tiers?

What statistical or machine learning methods do companies like Lending Club use to segment their customer base into loan grades A1-G5? What would a reasonable partitioning method look like after ...
2
votes
1answer
386 views

Calculating total probability of event in a time period given probabilities of it happening at different time intervals during that time period

I have time intervals and probabilities for an event to occur in these time intervals. I want to calculate the probability of the event happening once or more during the entire period, that is to say ...