Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms

Filter by
Sorted by
Tagged with
111
votes
4answers
30k views

Assessing approximate distribution of data based on a histogram

Suppose I want to see whether my data is exponential based on a histogram (i.e. skewed to the right). Depending on how I group or bin the data, I can get wildly different histograms. One set of ...
79
votes
7answers
30k views

What is the benefit of breaking up a continuous predictor variable?

I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model. It seems to me that by binning the variable we lose ...
22
votes
3answers
2k views

Benefits of using QQ-plots over histograms

In this comment, Nick Cox wrote: Binning into classes is an ancient method. While histograms can be useful, modern statistical software makes it easy as well as advisable to fit distributions to ...
21
votes
2answers
7k views

When should we discretize/bin continuous independent variables/features and when should not?

When should we discretize/bin independent variables/features and when should not? My attempts to answer the question: In general, we should not bin, because binning will lose information. Binning is ...
18
votes
2answers
3k views

Impact of data-based bin boundaries on a chi-square goodness of fit test?

Leaving aside the obvious issue of the low power of the chi-square in this sort of circumstance, imagine doing a chi-square goodness of test for some density with unspecified parameters, by binning ...
14
votes
3answers
5k views

Best way to put two histograms on same scale?

Let's say I have two distributions I want to compare in detail, i.e. in a way that makes shape, scale and shift easily visible. One good way to do this is to plot a histogram for each distribution, ...
11
votes
2answers
10k views

Optimal Binning with respect to a given response variable

I'm looking for optimal binning method (discretization) of a continuous variable with respect to a given response (target) binary variable and with maximum number of intervals as a parameter. example:...
10
votes
5answers
4k views

Why should binning be avoided at all costs?

So I've read a few posts about why binning should always be avoided. A popular reference for that claim being this link. The main getaway being that the binning points (or cutpoints) are rather ...
10
votes
5answers
2k views

Interpretation of Bayes Theorem applied to positive mammography results

I'm trying to wrap my head around the result of Bayes Theorem applied to the classic mammogram example, with the twist of the mammogram being perfect. That is, Incidence of cancer: $.01$ ...
10
votes
2answers
9k views

How to 'intelligently' bin a collection of sorted data?

I am trying to intelligently bin a sorted collection. I have a collection of $n$ pieces of data. But I know that this data fits into $m$ unequally sized bins. I don't know how to intelligently choose ...
9
votes
2answers
6k views

Number of bins when computing mutual information

I want to quantify the relationship between two variables, A and B, using mutual information. The way to compute it is by binning the observations (see example Python code below). However, what ...
9
votes
2answers
5k views

Doane's formula for histogram binning

I'm implementing various algorithms to estimated the best number of bins to use for histograms. Most of the ones I am implementing are described on the Wikipedia "Histogram" page in the section "...
8
votes
2answers
3k views

What is the justification for unsupervised discretization of continuous variables?

A number of sources suggest that there are many negative consequences of the discretization (categorization) of continuous variables prior to statistical analysis (sample of references [1]-[4] below). ...
8
votes
3answers
558 views

Can the 'bin size' in a histogram be thought of as a regularity constraint?

When thinking about a histogram as an estimate of the density function, is it reasonable to think of the bin size as a parameter that constrains the local structure of that function? Also, is there a ...
8
votes
2answers
30k views

How to find average and median age from an aggregated frequency table

I am using excel and I am trying to find both the average age and median age. I have two columns. 1 for the category and the other for the number of people in each category. ...
8
votes
1answer
184 views

How can I compare my model to a technically invalid model?

I've created nice little nonlinear model relating survival probability to length in salmon. I fit it assuming binomial errors and minimizing the negative log likelihood. I've been asked to compare it ...
8
votes
3answers
2k views

Is binning data valid prior to Pearson correlation?

Is it acceptable to bin data, calculate the mean of the bins, and then derive the Pearson correlation coefficient on the basis of these means? It seems a somewhat fishy procedure to me in that (if you ...
7
votes
3answers
2k views

Estimate of parameter of exponential distribution with binned data

I have the following data, which can be modeled by exponential distribution ...
7
votes
4answers
352 views

What is the mathematically rigorous definition of chunky data?

When in the workplace, certain measurement-taking devices are subject to different numerical accuracy; in some cases, the accuracy can be pretty weak (i.e., to one or two significant values only). ...
7
votes
1answer
2k views

Should we bin continuous variables?

I know this has been asked before, and I have read through the responses to the earlier queries related to binning continuous variables. I do understand that generally we should avoid binning, given ...
7
votes
1answer
1k views

How Do You Choose The Number of Bins To Use For A Chi-Squared GOF Test?

I'm working on developing a physics lab about radioactive decay, and in analyzing sample data I've taken, I ran into a statistics issue that surprised me. It is well known that the number of decays ...
7
votes
3answers
1k views

Interpolating binned data such that bin average is preserved

Say I have this binned data as input. The average value $\bar{y}_i$ is given for each successive $\Delta x_i$ interval. For simplicity, let's assume sampling density is uniform within each bin. Now I ...
6
votes
3answers
5k views

Is it valid to derive a mean from categorical data?

I am working on a study to quantify average working hours for doctors. However, when I leave it empty for respondents to fill up, it remains unfilled. Changing it into categories as above yield ...
6
votes
3answers
4k views

Logistic regression: categorical predictor vs. quantitative predictor

Why is it the case that when I run logistic regression with one categorical predictor, my regression is not significant whereas if I run the logistic regression with the same variable except it is ...
6
votes
2answers
37k views

Binning By Equal-Width

I have a dataset: 5, 10, 11, 13, 15, 35, 50 ,55, 72, 92, 204, 215 The formula for binning into equal-widths is this (as far as I know) $$width = (max - min) / N$...
6
votes
1answer
144 views

Where can I find more materials on 'binning' after PCA?

I am supposed to write a literature review on a particular paper for my University and I am lost after reading the main paper I am supposed to read. The link to the paper is here. The paper is from ...
5
votes
1answer
1k views

Clustering data into bins of variable sizes

I'd like to build a model (in R or excel) that takes in large amount of linear data and segments it into "bins". The linear data is an attribute that reflects what condition that section/record is at. ...
5
votes
1answer
986 views

Post-hoc power size calculation

I have, probably, a simple problem. I've finished analysing the results of an observational prospective study conducted in our unit. In this study I evaluated if a specific biomarker is independently ...
4
votes
1answer
3k views

Rationale for the use of Regressogram (Bin-Smooth)

I am taking a class in data mining and we have recently been introduced to bin-smoothing in regression analysis but i cannot seem to understand the usefulness of this method nor how the method works ...
4
votes
2answers
8k views

Best way to bin continuous data

I have a data frame with 1 vector of integers and 1 as a character factor like so: I have created a linear model that shows a relationship between age and party affiliation. I now want to determine ...
4
votes
2answers
3k views

Derive percentiles from binned data

The question below was asked on a sister site (Stack Overflow) back in 2010 by a user still active there (to me it seems more suitable here, for example quite similar to 21422): I have a bunch of ...
4
votes
1answer
330 views

Why would you band continuous variables in GLM?

Other than computational ease/requirements - are there reasons to band continuous variables? It seems to be a thing at my work place where everyone would split continuous data into 20-ish categories ...
4
votes
1answer
4k views

Why Binning Variables in Predictive Analytics?

Lot of discussion in CrossValidated focuses on optimal binning methods, binning example etc. But I am trying to figure out what are the scenarios that I have to bin variables whereas it's better idea ...
3
votes
2answers
195 views

A data-independant transformation to discretize a range of values non-uniformly

I am sure this is trivial, but I am looking for a transformation that nonuniformly discretizes all values of a range into several bins. The bins should be variant and I'd like them to be smaller ...
3
votes
2answers
5k views

Should the final R glm include only significant levels of factors

I am running a glm in R on data with quite many predictors (~50), both initially continuous and factors. The response is binary and the volume of the data is OK (~100K rows), in order to model non-...
3
votes
1answer
3k views

Characteristic of good binning for weight of evidence algorithm

I am using logistic regression for classification purpose. For reduction of features and better precision I am using Weight of evidence technique. Also I need to use python for this. As there is no ...
3
votes
2answers
242 views

How can I convert the result of a logistic regression into a set of classes?

I have a logistic regression $\text{logit} ( p_i ) = \beta_0 + \sum_j \beta_j x_{ij}$ with a binary response variable that I'd like to form a kind of scorecard from. By creating a scorecard I simply ...
3
votes
1answer
129 views

Measure of central tendency for periodic variable (hour of day)

I have a dataset that shows, for each group, the number of times a certain action was completed during each hour of the day. ...
3
votes
2answers
46 views

How to model gender specific values/variables as a predictor variable in the regression model?

My research question is to check whether the Body fat is associated with Hypertension onset. I am using Body fat as a categorical variable (i.e according to the value of body fat, the person will be ...
3
votes
1answer
581 views

Mutual info via binning gives non-zero results for independent variables

I'm trying to calculate mutual information in Python, using numpy. My implementation so far is: ...
3
votes
1answer
250 views

Multi Categorical Features vs multiple Features for categories

Say I am discretizing continuous data based on percentiles. (I realize this is generally frowned upon, but I am doing this for the sake of experiment) I am trying different percentiles, eg breaking ...
3
votes
0answers
117 views

How to compute optimal binning for two histograms

I am plotting two histograms on top of one another (using matplotlib, but that is tangential to my question). My current approach is to compute the mean of the optimal bin widths for each histogram ...
3
votes
0answers
124 views

treatment for factors with many levels [duplicate]

I'm running a predictive model and I have one possible predictive variable that is a factor and has more than 800 levels. I tried to reduce it running ctree in R (with the variable as the only ...
3
votes
0answers
2k views

Combining errors in a histogram (binned data)

I'm processing some data that requires binning before it goes through a regression algorithm. The script is in Python and uses the Numpy histogram function, but ...
2
votes
2answers
421 views

The value of an Effect Size

I calculated a Cohen's d value of d= -2.1. I understand that there are small, medium, and large effect sizes. But in my case the d value is negative? Would it still be considered large since abs(-2.1)...
2
votes
3answers
4k views

Counts of binned data by group

I have continuous data $A$ and categorical data $O$. I need counts of $A$ in bins by group $O$. I'm working in R. I know how to bin data (using cut2) and how to ...
2
votes
1answer
380 views

Exact multinomial goodness-of-fit test as a normality test

We have a practical real-life problem in an open source Linux related project. And I would like to hear an expert review/opinion about the way we are trying to solve this problem. It's been more than ...
2
votes
1answer
89 views

Regression to Classification and back to Regression

Is it reasonable to transform regression problem into classification by binning target variable into classes and construct regression curve separately on each class?\ Precisely, if my goal is to ...
2
votes
1answer
418 views

Cox-Proportional Hazards Survival Curve has too many lines - can binning the continuous variable help?

I am doing survival analysis on some continuous variables and am finding that some of my plots are difficult to interpret because there are too many lines. Here is an example: I am interested in ...
2
votes
1answer
20 views

How do you test for bias in a circular reference plane?

I've been trying to get my head around how to do hypothesis testing for a circular scale in hypothesis testing, but I am having a lot of trouble. I know well how to test for linear scales, but when it ...