Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms

Filter by
Sorted by
Tagged with
0
votes
0answers
66 views
+50

Binning correlated variables after model fitting

Having read Frank Harrel's list of problem caused by binning continuous variables (http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous), I understand that binning should be avoided for model ...
1
vote
2answers
38 views

Is binning of continuous data always bad for statistical tests? [duplicate]

I was always thinking that binning of data if data is naturally continuous is bad. However, here is the case. The goal of study was to find if there is an association between a biomarker and disease ...
1
vote
1answer
78 views

Regression to Classification and back to Regression

Is it reasonable to transform regression problem into classification by binning target variable into classes and construct regression curve separately on each class?\ Precisely, if my goal is to ...
2
votes
2answers
50 views

good number of bins for logarithmic bin width

I was wondering how to estimate a good number of bins for my histogram. I know quite certainly that my data is well approximated by a LogNormal distribution. Previous studies have used logarithmic ...
2
votes
1answer
20 views

How do you test for bias in a circular reference plane?

I've been trying to get my head around how to do hypothesis testing for a circular scale in hypothesis testing, but I am having a lot of trouble. I know well how to test for linear scales, but when it ...
0
votes
1answer
30 views

How to make SalePrice as a discrete value?

The target variable, Saleprice originally is a continuous value. I calculated ...
0
votes
0answers
20 views

(How) Does different number of categories affect correlation?

I work with obesity in cats and many studies use a body condition score (BCS) to assess obesity. This is a somewhat subjective measure of how much fat covering an animal has. There are two commonly ...
1
vote
0answers
27 views

Combining categories by Weight of Evidence

When calculating Information Value and Weight of Evidence, it's possible to draw a chart of WoE for each variable to study its effect on the state of the target variable. Now, I know it's possible to ...
0
votes
0answers
12 views

Comparing means in bins of different sample sizes

Suppose I have the following 2 plots showing the mean of a variable on a 2-d spatial grid, as well as the occurrence histogram (see bottom). Is there a specific way to compare these means, even ...
0
votes
0answers
23 views

problems in plotting decimal value distribution with bin width normalization in r using hist()

I am plotting data distribuiton. I get the expected plot with data which are only integer numbers. But I didn't get appropriate plot for decimal data sets. The code with integer numbers data as ...
1
vote
0answers
30 views

What is the name for bins derived from Quantiles? [duplicate]

It's so easy to talk about "deciles" as if they were groups of observations that fall between the actual deciles, i.e., "any of the nine values that divide the sorted data into ten equal parts." But ...
2
votes
1answer
24 views

Equivalent of Kaplan Meier for an unbounded number of sets

I have used Kaplan-Meier method several times before when I compared how group $A$ survived compared to group $B$ through a period of (say) 5 years. Now I face a somewhat different scenario: I have ...
10
votes
5answers
3k views

Why should binning be avoided at all costs?

So I've read a few posts about why binning should always be avoided. A popular reference for that claim being this link. The main getaway being that the binning points (or cutpoints) are rather ...
0
votes
3answers
266 views

statsmodels logistic regression with binned variables has large coefficients and standard error for some variables

I'm fitting a logistic regression (binary) using Python's statsmodels, and here's a snippet of summary from the model: I have noticed that the large coefficients ...
0
votes
1answer
114 views

Optimal multivariate binning where the cut-points must be the same for all observations

I have a large data set with many discrete and continuous variables. All the variables are present in every observation. I want to explain (the log of) one continuous variable using all the other ...
0
votes
0answers
49 views

Statistical test for binned proportion data in JMP / MATLAB

I have been trying to determine the proper statistical test for comparing binned proportional data between groups. My data set is individual subjects (4-5 per group) with 76-165 cells per individual. ...
1
vote
0answers
92 views

Data binning with error bars

I've got some measured data where each data point has an uncertainty associated with it i.e, Data $\mathbf{x} = (x_1,x_2,...,x_n)$ with uncertainties $\mathbf{\alpha}_x = (\alpha_1,\alpha_2,...,\...
1
vote
0answers
170 views

How to fit a distribution to binned values that come from administrative data?

Fitting a distribution to data (e.g. with maximum likelihood), or testing goodness of fit (e.g. with Kolmogorov-Smirnov) assumes that the data are randomly drawn from a population. But what if the ...
1
vote
0answers
32 views

How creating bins for a numeric feature can enables the model to learn nonlinear relationships within a single feature?

I understood How binning of numerical feature would help build correlations between the feature & the predictor. For example For a regression problem, we can bucketize "population" feature into ...
2
votes
1answer
97 views

Splitting range of independent variable to maximize prediction within the subranges

I have a dataset with two independent variables $X,Z \in \mathbb{R}$ and a dependent variable $Y \in \mathbb{R}$. This dataset has the following characteristics: given some number $z$ and a "small" $...
1
vote
1answer
325 views

Scott's and Freedman–Diaconis rules of the thumb for selecting bin width - disatvantages

Scott's and Freedman–Diaconis rules of the thumb are based on the following formula: In the article here it is stated that: While these appear to be useful estimates for unimodal densities ...
1
vote
1answer
69 views

Should one binarize qualitative variables before applying a random forest?

On which of theses two kinds of sample would a Random Forest (and more precisely sklearn RandomForest algorithm) give the best results ? (Y and other_features are continuous numerical variables, and ...
1
vote
0answers
83 views

Discretize values/binning for missing data

I’m running an experiment where pairs make ratings after answering questions. There are 15 minute intervals where they attempt to answer 12 questions, but each pair makes it through a different number ...
2
votes
0answers
30 views

Whether and how to bin data beyond a certain threshold

I've seen many admonishments against binning and agreed with them in general, but I wonder it's advisable to do so under certain circumstances. To adapt an example from my actual project, let's say ...
0
votes
1answer
494 views

Logarithmic binning and log-normal distribution

I've an Italian cities dataset. It's similar to those British ones used in literature, but has some differences, though. I decided to perform a logarithmic binning to avoid noise on the right end of ...
1
vote
1answer
30 views

Dichtomising continious treatment variables

In the past, I have repeatedly seen studies that dichtomize continuous treatment variables into a binary treatment dummy. I feel that this cannot be good practice. we loose important information we ...
2
votes
0answers
68 views

R Package for Optimal binning With Restrictions

Does anyone know about a R Package like SMBinning that can make Optimal Binning for WoE analysis where you can impose restrictions like minimum x number of Target in each bin monotome trend (WoE is ...
0
votes
0answers
33 views

How to correct make binning

I have a question regarding how to make binning correctly. I have 500 rows with different values (wages) which vary a lot. For example: in row number 20 the wage is \$600, whereas in 40 the wage is \...
0
votes
1answer
53 views

Why is it useful to have the same amount of data in each level of a categorical variable?

Something my lecturer said but I can't find why this is the case. I have to make a continuous variable into a categorical one, and the data is left skewed. Is it more important to have equal ranges or ...
2
votes
1answer
125 views

Learning a continuous model from binned data

A very similar question has been asked before, but it didn't get a real answer. Background I would like to develop a probability model for a continuous, ratio-scale random variable $Y$. Let's say it ...
0
votes
0answers
99 views

Distance Between 1-D Histograms and Binning

In order to calculate the distance between two 1-D histograms, they must have the same number of bins. I'm wondering if it's preferable to have more or fewer bins for such calculation. For example, ...
8
votes
3answers
483 views

Can the 'bin size' in a histogram be thought of as a regularity constraint?

When thinking about a histogram as an estimate of the density function, is it reasonable to think of the bin size as a parameter that constrains the local structure of that function? Also, is there a ...
2
votes
0answers
202 views

What is the correct way to modify the bin-counts given a threshold for a chi-squared test?

When performing a chi-square test, one inputs the expected counts (via integrating probability distribution over respective bin bounds) and observed counts into the chi-square formula (denoted below). ...
1
vote
0answers
516 views

Why do these two logarithmic binning methods give different results? (in R)

Problem I am working on a problem involving logarithmic binning. I have a dataset consisting of individuals that have a size value and a production value. I am using an algorithm to logarithmically ...
1
vote
0answers
46 views

How should I formulate the problem of “binning” many millions of points into thousands of bins?

I'm working on a type of string matching problem. In this problem, I have ~10k strings in set A and ~25 million strings in set B. For each string in set B, I want to find the string in set A which is ...
0
votes
0answers
782 views

Comparing the integral of a histogram and its distribution function

I asked a question here but I did a rather poor job of explaining my problem the question was poorly formed. If I have some histogram with $N_{total}$ number of total data points, and $N_{bins}$ ...
3
votes
0answers
116 views

How to compute optimal binning for two histograms

I am plotting two histograms on top of one another (using matplotlib, but that is tangential to my question). My current approach is to compute the mean of the optimal bin widths for each histogram ...
6
votes
1answer
123 views

Where can I find more materials on 'binning' after PCA?

I am supposed to write a literature review on a particular paper for my University and I am lost after reading the main paper I am supposed to read. The link to the paper is here. The paper is from ...
7
votes
1answer
879 views

How Do You Choose The Number of Bins To Use For A Chi-Squared GOF Test?

I'm working on developing a physics lab about radioactive decay, and in analyzing sample data I've taken, I ran into a statistics issue that surprised me. It is well known that the number of decays ...
0
votes
1answer
223 views

Defining bin sizes for plotting histograms [duplicate]

Perhaps a basic question, but is there a method or definition of how to choose the optimum bin size for some data with the intent to plot as a histogram? At present the best option I can think of is ...
1
vote
0answers
690 views

Binned residuals in logistic regression

I noticed that it's very hard to find something on binned residuals explained in a easy way except for this. This is my first time approaching to binned residuals. I understood that is not useful ...
0
votes
1answer
53 views

Is it incorrect to interpolate between known probabilities for binned data?

I have 7 bins of binary data according to a variable x as follows: x1: N=1000, success = 324, failure=676 x2: N=1000, success = 444, failure= 556 .... .... x7: N=...., success = ..., failure = ......
0
votes
1answer
222 views

Normal distribution applied to oil prices

I'm taking an introduction to statistics course and in attempt to reinforce what I have learned I have fit normal distributions to oil open,high,low,close prices over past 10 years where the number of ...
1
vote
1answer
135 views

R: Automatically group the insignificant dummy levels and re-fit the model

I am running a reg model with Weekdays as my dummy variables so I can find the weekday effects to the output metric. The picture below shows the results from this regression, and only Sunday is ...
0
votes
0answers
988 views

Converting a continuous variable to categorical

I have several continuous predictor variables and one binary outcome variable. One of these predictor variables has the following description using the Hmisc package: ...
0
votes
1answer
452 views

Discretizing/Binning Continuous Variable by Continuous Response

I am trying to discretize a continuous variable that does not follow a linear relationship with our response variable. I am trying to find the optimal way to discretize this variable to better ...
1
vote
3answers
1k views

Creating a Predictive Model with Binned Data

I have a health dataset with the number of drinks per month someone consumes, and many other variables that are binned. For example, 1: income less than \$10000, 2=income less than \$20000, and so on. ...
1
vote
0answers
49 views

How are various rules for determining frequency distribution binwidth derived?

I've been looking up the problem of deciding appropriate binwidths for histograms and here's my broad-level understanding so far: If we have $n$ data points, we assume that they're realizations of $n$...
0
votes
0answers
28 views

Given a dataset and no information about its distribution, how to determine bin size?

Let's say I have a $k$-vector of numeric data $X \in \mathbb{R}^k$, and I want to plot a frequency distribution of all the data points in the vector. I've tried searching high and low for a method (...
2
votes
2answers
1k views

RandomForest( ) more than 32factor levels [closed]

I used randomForest( ) function in R. But R can't deal variables that more 32factor levels. But my factor variables are very important thing. So I want to use this variable. Then, how to deal this ...