We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms

Filter by
Sorted by
Tagged with
1
vote
0answers
194 views

How to compare binned CDFs?

We want to compare binned CDFs from sampled data. Each CDF is 101 numbers representing percentiles, i.e., $P(x<a_k)=k/100$ where $k=0..100$. The sample size is 1-3k (unknown at comparison time ...
9
votes
2answers
5k views

Doane's formula for histogram binning

I'm implementing various algorithms to estimated the best number of bins to use for histograms. Most of the ones I am implementing are described on the Wikipedia "Histogram" page in the section "...
2
votes
0answers
71 views

References for the effect of binning methods?

There are two main ways to bin a numerical signal into discrete categorical values: Quintile Binning: Each Bin will have the same size Histogram Binning: A normal distribution will be used to bin the ...
3
votes
1answer
127 views

Measure of central tendency for periodic variable (hour of day)

I have a dataset that shows, for each group, the number of times a certain action was completed during each hour of the day. ...
2
votes
1answer
83 views

How to compute the central tendency for such a distribution?

Consider an annular region of space. We divide this annulus into four equal regions and we take some measurements in each of the four sectors. Imagine we are measuring something like a particle's ...
1
vote
0answers
513 views

Binning Logistic Regression Results by Output in R

I have a logistic regression in R whose goal is to predict the probability of default on some test data. ...
2
votes
1answer
411 views

Cox-Proportional Hazards Survival Curve has too many lines - can binning the continuous variable help?

I am doing survival analysis on some continuous variables and am finding that some of my plots are difficult to interpret because there are too many lines. Here is an example: I am interested in ...
0
votes
1answer
59 views

How to handle continous data with several peaks

I'm running a simulation that produces continuous data distributed in 4 to 6 peaks. Each peak is roughly normally distributed. I'd like to detect each peak mean value and relative weight. Right now I'...
7
votes
4answers
348 views

What is the mathematically rigorous definition of chunky data?

When in the workplace, certain measurement-taking devices are subject to different numerical accuracy; in some cases, the accuracy can be pretty weak (i.e., to one or two significant values only). ...
4
votes
1answer
4k views

Why Binning Variables in Predictive Analytics?

Lot of discussion in CrossValidated focuses on optimal binning methods, binning example etc. But I am trying to figure out what are the scenarios that I have to bin variables whereas it's better idea ...
3
votes
1answer
564 views

Mutual info via binning gives non-zero results for independent variables

I'm trying to calculate mutual information in Python, using numpy. My implementation so far is: ...
10
votes
5answers
2k views

Interpretation of Bayes Theorem applied to positive mammography results

I'm trying to wrap my head around the result of Bayes Theorem applied to the classic mammogram example, with the twist of the mammogram being perfect. That is, Incidence of cancer: $.01$ ...
2
votes
0answers
16 views

How to test if $I(\theta)\propto f(\theta)$?

How do I test if a finite resource is distributed across a population proportional to the density function for that population? Say $I(\theta)$ denotes resource allocation across $\theta$ (a known ...
9
votes
2answers
6k views

Number of bins when computing mutual information

I want to quantify the relationship between two variables, A and B, using mutual information. The way to compute it is by binning the observations (see example Python code below). However, what ...
2
votes
2answers
2k views

optimal binning in R

SPSS has an optimal binning function that helps categorizing into meaningful intervals continuous predictors when a binary response variable exists. I was looking for an equivalent function in R but I'...
1
vote
0answers
702 views

Supervised Binning with Naive Bayes

Context. I am working on a model to predict "churn". Subscription service where users pay a monthly fee to access the service. We would like to predict which accounts are likely to cancel or "churn". ...
0
votes
1answer
707 views

How to perform binning in order to discretize continuous features for feature selection in R?

If I wanted to use uncertainty measures e.g. information gain for feature selection continuous features need to be discretized. How can I do this in R? Thank you for your help.
4
votes
2answers
8k views

Best way to bin continuous data

I have a data frame with 1 vector of integers and 1 as a character factor like so: I have created a linear model that shows a relationship between age and party affiliation. I now want to determine ...
2
votes
0answers
1k views

converting continous variable to discrete while maximizing gini

I have a continuous independent variable which is used to explain the dependent binary variable in logistic regression. The model user's requirement is to group this continuous variable to 6 bins. I ...
0
votes
0answers
478 views

Binning a continuous feature for a SVM

I am using a support vector machine (SVM) for binary classification. One of my features is continuous: each item has an attribute $x$ that is a real number. For various reasons (e.g., because I ...
1
vote
0answers
495 views

Comparing two sets of equal width binned data

I have two distinct sets of ~1500 intervals of differing lengths - let’s call them Interval Set 1 and Interval Set2. I have ...
0
votes
1answer
79 views

How to calculate third variable dependence on x,y variables and visualise with heatmap using binning?

I have a dataframe with an X and Y column and a third column with an additional variable (let's call it "ABC"). I would like to create a heatmap that visualizes how ABC depends on the X and Y ...
1
vote
0answers
120 views

Find bins for observations where each bin has similar mean

I have 100000 observations with two variables on each, age on the range of 18-80 and interactions on the range of 1-1500. I want ...
5
votes
1answer
973 views

Post-hoc power size calculation

I have, probably, a simple problem. I've finished analysing the results of an observational prospective study conducted in our unit. In this study I evaluated if a specific biomarker is independently ...
2
votes
2answers
853 views

Create bins for lognormal data for cluster analysis

I have a series of dollar amounts that are highly right skewed, but are roughly log-normal. I want to put this grouped dollar amount as a predictor variable into a latent class cluster analysis. In ...
2
votes
1answer
92 views

Dichotomizing continuous variables for an EXPERIMENT not a survey

I've aware of the many issues/pitfalls with splitting continuous variables in survey research. Nothing I have read addresses whether doing so to create groups which will then be assigned to tx ...
7
votes
1answer
2k views

Should we bin continuous variables?

I know this has been asked before, and I have read through the responses to the earlier queries related to binning continuous variables. I do understand that generally we should avoid binning, given ...
2
votes
1answer
118 views

How to categorize count data

I have count data (basically histone modification data). The counts represents number of reads falling into each genomic region. I need to divide my data into $\text{low|medium|high}$ based on the ...
0
votes
1answer
51 views

Using the average as a cut off to group data and compare groups?

I measured cell size of human muscle and I wanted to examine the effect of age on the parameter. I graphed my data as a scatter plot and calculated Pearson's correlation coefficient but then also ...
4
votes
2answers
3k views

Derive percentiles from binned data

The question below was asked on a sister site (Stack Overflow) back in 2010 by a user still active there (to me it seems more suitable here, for example quite similar to 21422): I have a bunch of ...
5
votes
1answer
1k views

Clustering data into bins of variable sizes

I'd like to build a model (in R or excel) that takes in large amount of linear data and segments it into "bins". The linear data is an attribute that reflects what condition that section/record is at. ...
4
votes
1answer
3k views

Rationale for the use of Regressogram (Bin-Smooth)

I am taking a class in data mining and we have recently been introduced to bin-smoothing in regression analysis but i cannot seem to understand the usefulness of this method nor how the method works ...
2
votes
1answer
999 views

Can binning a continuous predictor or DV variable improve large data sets fit?

I read that averaging and binning a continuous predictor variable is in general a bad idea because it's always better to fit the continuous relationship through splines, poly and all of that. Sure, I ...
1
vote
1answer
760 views

Binning method: looking for an example

I heard and read several times of the use of 'binning' methods to estimate the uncertainty and the auto-correlation time of a sample generated by MCMC but I can't find a textbook example of it being ...
1
vote
0answers
71 views

How to select an error/weight for two-dimensional binned data?

Beginning with noisy data vectors $\mathbf{x}$ and $\mathbf{y}$, I have binned the data to vectors $\mathbf{x}_b$ and $\mathbf{y}_b$ of length $N_b$ with fixed linear ($\mathbf{x}_b^i - \mathbf{x}_b^{...
1
vote
0answers
636 views

selecting the bins for extremely skewed data

I have a data that exhibits nearly a power law distribution, and I want to know a good binning technique to summarize the statistics. For example consider the following data: $$ \begin{array}{rr} \...
1
vote
0answers
232 views

Is binning a skewed Likert scale variable justifiable?

I have recently advised some colleagues on the malpractice of binning a continuous variable, which was used in order to put it as a covariate in a regression model and retained as a significant ...
2
votes
2answers
5k views

Should the final R glm include only significant levels of factors

I am running a glm in R on data with quite many predictors (~50), both initially continuous and factors. The response is binary and the volume of the data is OK (~100K rows), in order to model non-...
8
votes
3answers
2k views

Is binning data valid prior to Pearson correlation?

Is it acceptable to bin data, calculate the mean of the bins, and then derive the Pearson correlation coefficient on the basis of these means? It seems a somewhat fishy procedure to me in that (if you ...
2
votes
0answers
544 views

A good alternative to data binning?

I read many times that data binning of continuous variables is a very bad idea. For instance, let's take something like heart rate and let's define the following 2 bins: (125 - 135), (136 - 145) ...
1
vote
1answer
470 views

How to bin a quantitative covariate for multiple regression?

I do a multiple regression in which spacing between lines is one of four variables. However, this spacing between lines varies so I have grouped them into five classes designated with numbers 1 ...
2
votes
1answer
873 views

Jensen-Shannon divergence for finite samples

I have two finite samples $s_1$ and $s_2$ and two distributions $p_1(s_1)$ and $p_2(s_2)$ that are associated to these samples. I'm essentially interested to measure the distance or similarity between ...
1
vote
2answers
632 views

One way analysis of continuous variables for classification (Visualising)

I had trouble coming up with a title, so hopefully I can explain it better here. I'm working on a classification problem and I'm doing some pre-analysis of variables. I'm looking for some nice ways ...
18
votes
2answers
3k views

Impact of data-based bin boundaries on a chi-square goodness of fit test?

Leaving aside the obvious issue of the low power of the chi-square in this sort of circumstance, imagine doing a chi-square goodness of test for some density with unspecified parameters, by binning ...
0
votes
1answer
2k views
1
vote
1answer
240 views

MLEs of Poisson lambda values

I have sample data that I expect to contain values from at least several Poisson distributions (set around various lambda values). Some of these lambda values are nicely spaced, leading to what are ...
1
vote
0answers
77 views

Should I bin my continuous input variables for neural networks? [duplicate]

And what is the optimal method to do this? Is there a package for this in R?
1
vote
0answers
652 views

Data Exploration - How To Bin Data?

There are 1100 data points split into about 35 discrete groups and distributed towards the left. After running a logistic regression which failed, then running it again by only predicting based on ...
2
votes
0answers
462 views

On the uniform convergence of relative frequencies of events to their probabilities

I have read the article by Vapnik, Chervonenkis "On the uniform convergence of relative frequencies of events to their probabilities" Theory of Probability and Its Applications, vol XVI, n. , 1971. ...
0
votes
0answers
882 views

Equal frequency vs. equal weight discretization

How does one choose between the two? How do you decide which one is more appropriate for your data? I have a numeric data and I would like to discretize it using one of these. I know what they do ...