Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms

Filter by
Sorted by
Tagged with
111
votes
4answers
30k views

Assessing approximate distribution of data based on a histogram

Suppose I want to see whether my data is exponential based on a histogram (i.e. skewed to the right). Depending on how I group or bin the data, I can get wildly different histograms. One set of ...
0
votes
0answers
33 views

How to correct make binning

I have a question regarding how to make binning correctly. I have 500 rows with different values (wages) which vary a lot. For example: in row number 20 the wage is \$600, whereas in 40 the wage is \...
0
votes
1answer
53 views

Why is it useful to have the same amount of data in each level of a categorical variable?

Something my lecturer said but I can't find why this is the case. I have to make a continuous variable into a categorical one, and the data is left skewed. Is it more important to have equal ranges or ...
2
votes
1answer
129 views

Learning a continuous model from binned data

A very similar question has been asked before, but it didn't get a real answer. Background I would like to develop a probability model for a continuous, ratio-scale random variable $Y$. Let's say it ...
0
votes
0answers
104 views

Distance Between 1-D Histograms and Binning

In order to calculate the distance between two 1-D histograms, they must have the same number of bins. I'm wondering if it's preferable to have more or fewer bins for such calculation. For example, ...
8
votes
3answers
539 views

Can the 'bin size' in a histogram be thought of as a regularity constraint?

When thinking about a histogram as an estimate of the density function, is it reasonable to think of the bin size as a parameter that constrains the local structure of that function? Also, is there a ...
6
votes
2answers
36k views

Binning By Equal-Width

I have a dataset: 5, 10, 11, 13, 15, 35, 50 ,55, 72, 92, 204, 215 The formula for binning into equal-widths is this (as far as I know) $$width = (max - min) / N$...
2
votes
0answers
223 views

What is the correct way to modify the bin-counts given a threshold for a chi-squared test?

When performing a chi-square test, one inputs the expected counts (via integrating probability distribution over respective bin bounds) and observed counts into the chi-square formula (denoted below). ...
1
vote
0answers
616 views

Why do these two logarithmic binning methods give different results? (in R)

Problem I am working on a problem involving logarithmic binning. I have a dataset consisting of individuals that have a size value and a production value. I am using an algorithm to logarithmically ...
1
vote
0answers
49 views

How should I formulate the problem of “binning” many millions of points into thousands of bins?

I'm working on a type of string matching problem. In this problem, I have ~10k strings in set A and ~25 million strings in set B. For each string in set B, I want to find the string in set A which is ...
7
votes
3answers
1k views

Interpolating binned data such that bin average is preserved

Say I have this binned data as input. The average value $\bar{y}_i$ is given for each successive $\Delta x_i$ interval. For simplicity, let's assume sampling density is uniform within each bin. Now I ...
0
votes
0answers
889 views

Comparing the integral of a histogram and its distribution function

I asked a question here but I did a rather poor job of explaining my problem the question was poorly formed. If I have some histogram with $N_{total}$ number of total data points, and $N_{bins}$ ...
78
votes
7answers
29k views

What is the benefit of breaking up a continuous predictor variable?

I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model. It seems to me that by binning the variable we lose ...
3
votes
0answers
117 views

How to compute optimal binning for two histograms

I am plotting two histograms on top of one another (using matplotlib, but that is tangential to my question). My current approach is to compute the mean of the optimal bin widths for each histogram ...
3
votes
1answer
3k views

Characteristic of good binning for weight of evidence algorithm

I am using logistic regression for classification purpose. For reduction of features and better precision I am using Weight of evidence technique. Also I need to use python for this. As there is no ...
6
votes
1answer
139 views

Where can I find more materials on 'binning' after PCA?

I am supposed to write a literature review on a particular paper for my University and I am lost after reading the main paper I am supposed to read. The link to the paper is here. The paper is from ...
1
vote
2answers
1k views

Binning Continuous Variables By Entropy From Binary Response (R) [closed]

I am working with at data set where the goal is to predict a binary response. I have a few continuous variable that I think would be beneficial to bin. I was reading this idea about entropy based ...
0
votes
1answer
258 views

Defining bin sizes for plotting histograms [duplicate]

Perhaps a basic question, but is there a method or definition of how to choose the optimum bin size for some data with the intent to plot as a histogram? At present the best option I can think of is ...
1
vote
0answers
748 views

Binned residuals in logistic regression

I noticed that it's very hard to find something on binned residuals explained in a easy way except for this. This is my first time approaching to binned residuals. I understood that is not useful ...
1
vote
1answer
1k views

Bins in Regression Discontinuity Designs

Lee and Lemieux (p. 31, 2009) suggest the researcher to also present graphs while doing Regression discontinuity design analysis. They suggest the following procedure: "...for some bandwidth $h$, ...
22
votes
3answers
2k views

Benefits of using QQ-plots over histograms

In this comment, Nick Cox wrote: Binning into classes is an ancient method. While histograms can be useful, modern statistical software makes it easy as well as advisable to fit distributions to ...
21
votes
2answers
7k views

When should we discretize/bin continuous independent variables/features and when should not?

When should we discretize/bin independent variables/features and when should not? My attempts to answer the question: In general, we should not bin, because binning will lose information. Binning is ...
0
votes
1answer
57 views

Is it incorrect to interpolate between known probabilities for binned data?

I have 7 bins of binary data according to a variable x as follows: x1: N=1000, success = 324, failure=676 x2: N=1000, success = 444, failure= 556 .... .... x7: N=...., success = ..., failure = ......
0
votes
1answer
241 views

Normal distribution applied to oil prices

I'm taking an introduction to statistics course and in attempt to reinforce what I have learned I have fit normal distributions to oil open,high,low,close prices over past 10 years where the number of ...
1
vote
1answer
136 views

R: Automatically group the insignificant dummy levels and re-fit the model

I am running a reg model with Weekdays as my dummy variables so I can find the weekday effects to the output metric. The picture below shows the results from this regression, and only Sunday is ...
0
votes
1answer
487 views

Discretizing/Binning Continuous Variable by Continuous Response

I am trying to discretize a continuous variable that does not follow a linear relationship with our response variable. I am trying to find the optimal way to discretize this variable to better ...
7
votes
3answers
2k views

Estimate of parameter of exponential distribution with binned data

I have the following data, which can be modeled by exponential distribution ...
1
vote
0answers
52 views

How are various rules for determining frequency distribution binwidth derived?

I've been looking up the problem of deciding appropriate binwidths for histograms and here's my broad-level understanding so far: If we have $n$ data points, we assume that they're realizations of $n$...
0
votes
0answers
28 views

Given a dataset and no information about its distribution, how to determine bin size?

Let's say I have a $k$-vector of numeric data $X \in \mathbb{R}^k$, and I want to plot a frequency distribution of all the data points in the vector. I've tried searching high and low for a method (...
2
votes
2answers
1k views

RandomForest( ) more than 32factor levels [closed]

I used randomForest( ) function in R. But R can't deal variables that more 32factor levels. But my factor variables are very important thing. So I want to use this variable. Then, how to deal this ...
3
votes
0answers
124 views

treatment for factors with many levels [duplicate]

I'm running a predictive model and I have one possible predictive variable that is a factor and has more than 800 levels. I tried to reduce it running ctree in R (with the variable as the only ...
1
vote
0answers
77 views

Should I bin this continuous variable?

I understand that binning is generally frowned upon as you are "throwing away information". However, I'm not sure whether I should in this instance. I am running a logit regression. One of my ...
2
votes
0answers
60 views

How do loan companies set interest rate tiers?

What statistical or machine learning methods do companies like Lending Club use to segment their customer base into loan grades A1-G5? What would a reasonable partitioning method look like after ...
1
vote
0answers
502 views

To BIN or not to BIN a continuous data to get a Fragment Size Distribution?

I have a data set in excel of almost 6000 entries (quantitative and continuous => P(X=x)=0, I mean the possible values for my continuous random variable X are uncountably many). Each point will ...
2
votes
1answer
454 views

Calculating total probability of event in a time period given probabilities of it happening at different time intervals during that time period

I have time intervals and probabilities for an event to occur in these time intervals. I want to calculate the probability of the event happening once or more during the entire period, that is to say ...
2
votes
0answers
79 views

Given the number of Bins, what is the formula for estimating bin height in a 2D Histogram

Given the number of Bins, what is the formula for estimating bin height in a 2D Histogram in binormal distribution?
1
vote
0answers
111 views

Learning optimal histogram bins

I have a a data set containing of non-negative integer data for N subjects. In other words, each subject is represented by a vector of non negative integers (the vector length may vary from subject to ...
1
vote
0answers
618 views

Programmatically calculate which bin a value will fall into for a histogram

I'm trying to programmatically create a histogram. The number of bins is specified by the user. My issue in this problem is that I came up with what I thought was a reasonable way to determine which ...
3
votes
2answers
239 views

How can I convert the result of a logistic regression into a set of classes?

I have a logistic regression $\text{logit} ( p_i ) = \beta_0 + \sum_j \beta_j x_{ij}$ with a binary response variable that I'd like to form a kind of scorecard from. By creating a scorecard I simply ...
11
votes
2answers
10k views

Optimal Binning with respect to a given response variable

I'm looking for optimal binning method (discretization) of a continuous variable with respect to a given response (target) binary variable and with maximum number of intervals as a parameter. example:...
1
vote
0answers
305 views

supervised binning

I have continuous outcome variable and continuous independent variable. I am trying to bin the independent variable that maximizes homogeneity within bins based on the outcome and maximize separation. ...
0
votes
1answer
697 views

Calculate variance from mean and variance of bins [duplicate]

I have an unbounded process which generates normal distributed values (could be ints, but lets assume floating points for now). These values are put into bins of fixed size (e.g. bin 9 gets values in ...
2
votes
1answer
70 views

Analyzing data with a cumulative component

I've got a computer science background, but am new to any real statistics work. I've been asked to answer a question, and I'm looking for some guidance on how to approach it. I work in the food ...
2
votes
2answers
418 views

The value of an Effect Size

I calculated a Cohen's d value of d= -2.1. I understand that there are small, medium, and large effect sizes. But in my case the d value is negative? Would it still be considered large since abs(-2.1)...
2
votes
1answer
375 views

Exact multinomial goodness-of-fit test as a normality test

We have a practical real-life problem in an open source Linux related project. And I would like to hear an expert review/opinion about the way we are trying to solve this problem. It's been more than ...
6
votes
3answers
4k views

Is it valid to derive a mean from categorical data?

I am working on a study to quantify average working hours for doctors. However, when I leave it empty for respondents to fill up, it remains unfilled. Changing it into categories as above yield ...
4
votes
1answer
326 views

Why would you band continuous variables in GLM?

Other than computational ease/requirements - are there reasons to band continuous variables? It seems to be a thing at my work place where everyone would split continuous data into 20-ish categories ...
0
votes
1answer
441 views

How can I use clustering algorithms to bin highly skewed data process?

I have a large set of multi dimensional data.The data points are highly skewed and not smoothly distributed.I want to divide the data set to some finite number of bins.I have approached this problem ...
0
votes
0answers
2k views

Binning by boundaries

What happens in the situation where you have a value that is equi-distant to the upper and lower boundaries when binning by boundaries? Take the example {26,28,30,34} Does 30 get converted to 26 or ...
1
vote
1answer
217 views

Reference for known issues with histograms (binning, anchor point)

I'm looking for references on the known issues that arise when working with histograms, i.e.: the choice of the number of bins, and the choice of the origin point. The WP entry on Multivariate kernel ...